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Highlights 

Highlights of GAO-08-46, a report to 
congressional committees 



Why GAO Did This Study 

The Department of Homeland 
Security (DHS) established the 
Automated Commercial 
Environment (ACE) program to 
replace and supplement existing 
cargo processing technology. 
According to the fiscal year 2007 
DHS appropriations act, DHS is to 
develop and submit an expenditure 
plan for ACE that satisfies certain 
conditions, including being 
reviewed by GAO. 

GAO reviewed the plan to (1) 
determine whether the expenditure 
plan satisfies the legislative 
conditions, (2) determine the status 
of 15 open GAO recommendations, 
and (3) provide observations about 
the expenditure plan and DHS's 
management of the program. To 
address the mandate, GAO 
assessed plans and related 
documentation against federal 
guidelines and industry standards 
and interviewed the appropriate 
DHS officials. 



What GAO Recommends 



GAO is making recommendations 
to further strengthen ACE 
management and accountability by 
disclosing program information in 
quarterly reports to Congress 
related to unmet legislative 
conditions, open GAO 
recommendations, program 
changes, and risk management. 
DHS agreed with GAO's findings 
and recommendations and 
described actions that it has under 
way and planned to address them. 
Most of the described actions are 
consistent with GAO's 
recommendations. 



To view the full product, including the scope 
and methodology, click on GAO-08-46. 
For more information, contact Randolph C. 
Hite at (202) 512-3459 or hiter@gao.gov. 



INFORMATION TECHNOLOGY 

Improvements for Acquisition of Customs 
Trade Processing System Continue, but 
Further Efforts Needed to Avoid More Cost and 
Schedule Shortfalls 

What GAO Found 

The ACE expenditure plan satisfies many — but not all — of the legislative 
conditions specified in the fiscal year 2007 DHS appropriations act. 
Specifically, the plan (with related program documentation and officials' 
statements) complies with acquisition rules, requirements, guidelines, and 
management practices of the federal government; includes a DHS certification 
that an independent verification and validation agent is under contract; was 
reviewed and approved by DHS and the Office of Management and Budget 
(OMB); and was reviewed by GAO. In addition, it partially satisfies conditions 
for meeting the capital planning and investment control review requirements 
established by OMB in Circular A-ll (part 7), including information security, 
and for complying with the DHS enterprise architecture. 

DHS has implemented eight open GAO recommendations made during the 
past 4 years, including those related to performance measures and targets, 
independent verification and validation, cost estimation, and program 
reporting. Seven other recommendations made during this time are in the 
process of being implemented. With respect to these seven, DHS has taken 
steps to satisfy each, such as establishing an accountability framework, 
reducing overlap and concurrence among ACE releases, and completing a 
privacy impact assessment, and actions are under way or planned to more 
fully address them. 

GAO is making three new observations about the expenditure plan and the 
management of ACE. First, the program is taking needed steps to redefine 
requirements for several ACE releases because of limitations in the 
completeness of original requirements, but this redefinition is likely to 
introduce significant program schedule delays and cost increases. Second, the 
changes to ACE requirements have led to replacement of a key commercial 
product, but the new product carries the risk of negatively impacting user 
productivity. Third, the automated database used for managing ACE risks is 
incomplete and does not contain information needed to adequately inform 
program decisions. 

All told, DHS has continued to make progress on ACE, and the program is 
better positioned today for delivering promised capabilities and benefits than 
it has been in the past. Nevertheless, key program management practices 
relating to, for example, human capital management, requirements 
management, and risk management remain a challenge, and other 
management areas, such as information security and 

architecture alignment, continue to require attention. As a result, GAO sees 
major program schedule delays and cost overruns on the horizon. To improve 
ACE management and minimize exposure to risk, it is important for DHS to 
remain vigilant in its efforts to satisfy ACE legislative requirements, fully 
implement prior GAO recommendations, and keep Congress fully informed 
about the program's status, plans, and risk. 
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United States Government Accountability Office 
Washington, D.C. 20548 



October 25, 2007 

The Honorable Robert C. Byrd 
Chairman 

The Honorable Thad Cochran 
Ranking Minority Member 
Subcommittee on Homeland Security 
Committee on Appropriations 
United States Senate 

The Honorable David E. Price 
Chairman 

The Honorable Harold Rogers 
Ranking Minority Member 
Subcommittee on Homeland Security 
Committee on Appropriations 
House of Representatives 

The U.S. Customs and Border Protection (CBP), as part of the Department 
of Homeland Security (DHS), submitted to Congress in February 2007 its 
fiscal year 2007 expenditure plan for the Automated Commercial 
Environment (ACE) program pursuant to the Department of Homeland 
Security Appropriations Act, 2007. 1 

Begun in 2001, ACE is to replace and supplement existing cargo processing 
technology. The system is being developed and deployed in a series of 
increments to about 300 ports of entry and is expected to be fully deployed 
by the end of 2011. The goals of ACE include (1) supporting border security 
by enhancing analysis and information sharing with other government 
agencies and providing CBP with the means to decide before a shipment 
reaches the border what should be targeted because it is a security threat 
and what should be expedited because it complies with U.S. laws and (2) 
streamlining time-consuming and labor-intensive tasks for CBP personnel 
and the trade community through a national trade account and a single 
Web-based interface. 

As required by the appropriations act, we reviewed ACE's fiscal year 2007 
expenditure plan. Our objectives were to (1) determine whether the 
expenditure plan satisfies the legislative conditions, (2) determine the 



'Pub. L. No. 109-295 (Oct. 4, 2006). 
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status of 15 open GAO recommendations for ACE, 2 and (3) provide 
observations about the expenditure plan and DHS's management of the 
program. 

On July 26, 2007, we provided a briefing to the staffs of the Subcommittees 
on Homeland Security, Senate and House Committees on Appropriations 
on the results of our review. This report transmits those results. The full 
briefing, including our scope and methodology, is reprinted in appendix I. 



Compliance with 
Legislative Conditions 



The ACE expenditure plan — including related program documentation and 
program officials' statements — satisfies or partially satisfies the six 
legislative conditions specified in the appropriations act. Specifically, the 
plan satisfies the conditions that it (1) comply with the acquisition rules, 
requirements, guidelines, and systems acquisition management practices of 
the federal government; 3 (2) include a certification by the DHS Chief 
Information Officer that an independent verification and validation agent is 
currently under contract for the program; (3) be reviewed by the DHS 
Investment Review Board, the Secretary of DHS, and Office of Management 
and Budget (OMB); and (4) be reviewed by us. The plan and its supporting 
information partially satisfies the conditions that it (5) meet the capital 
planning and investment control review requirements established by OMB 
(including Circular A-ll, part 7) and that it (6) comply with the DHS 
enterprise architecture. For example, although DHS determined that ACE 
was aligned with the DHS enterprise architecture, the agency's analysis did 
not address key aspects of an architectural alignment, such as alignment 
with the architecture's data reference model. 



Status of GAO 
Recommendations 



DHS has implemented some, but not all, of the recommendations 
pertaining to ACE that we have made since 2003. These recommendations, 
along with their status, are summarized here. 



Two related open recommendations have been combined. 

3 We did not evaluate the compliance of the ACE program with respect to the Federal 
Acquisition Regulation or OMB directives, but we did evaluate its compliance with 
established best practices models that incorporate IT investment requirements for federal 
agencies. 
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Eight recommendations have been implemented. For example, DHS has 

• developed a range of ACE performance measures and targets needed to 
support an outcome-based, results-oriented accountability framework, 
including user satisfaction with ACE; 

• aligned ACE program goals, benefits, desired business outcomes, and 
performance measures; 

• addressed those legislative conditions associated with measuring ACE 
performance and results and employing effective independent 
verification and validation practices; and 

• ensured that future expenditure plans are based on cost estimates that 
are reconciled with independent cost estimates. 

The remaining seven recommendations are in the process of being 
implemented, as described here. 

• Recommendation: Define and implement an ACE accountability 
framework that fulfills several conditions, 4 to include ensuring the 
currency, relevance, and completeness of commitments made to 
Congress in expenditure plans and reporting in future expenditure plans 
the progress against commitments that were contained in prior 
expenditure plans. 

In progress. The program has established an accountability framework 
that is providing input for both the annual expenditure plan and quarterly 
congressional reports. However, as with prior expenditure plans, the fiscal 
year 2007 expenditure plan did not reflect the most current program 
information. For example, information on milestones, earned value 
management, and risks were about 4 months old when the plan was 
submitted to the appropriations committees in February 2007, and program 
commitments were no longer current. Further, while program officials 



4 This recommendation has multiple conditions, of which three are completed and two are in 
progress. A sixth condition of this recommendation — that the ACE accountability 
framework clearly and unambiguously delineate roles and responsibilities of the 
government and the prime contractor — was previously completed. See Information 
Technology: Customs Has Made Progress on Automated Commercial Environment 
System, but R Faces Longstanding Management Challenges and New Risks, GAO-06-580 
(Washington, D.C.: May 31, 2006). 
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continue to use the quarterly ACE status reports to provide the 
appropriations committees with more detailed information, these reports 
are generally submitted to Congress 3 to 4 months after the end of each 
quarter, thus limiting their currency and relevance as well. CBP and DHS 
officials told us that the delays were due to the DHS review and approval 
process and that they are exploring ways to accelerate the process. 

The 2007 expenditure plan did not adequately report on progress against 
previous plan commitments. For example, the plan did not (1) report 
progress against milestones in the fiscal year 2006 plan or explain why 
these milestones were not achieved, (2) report actual obligation or 
expenditure of funds relative to the planned uses of these funds in prior 
expenditure plans, or (3) address progress against the milestone dates for 
each stage of a release that was included in the prior year's plan. According 
to program officials, the quarterly congressional reports provide more 
current information on the program's progress against prior commitments. 
However, these reports have also not fully addressed the commitments 
made in prior expenditure plans. Program officials stated that they intend 
to start providing this information in the last quarterly report of each year. 

• Recommendation: Define measures and collect and use associated 
metrics for determining whether prior and future program management 
improvements are successful. 

Planned. The program office has made changes that are to improve overall 
program management and, according to its December 2006 quarterly report 
to Congress, the program office plans to measure the impact of future 
management improvements. Moreover, this report stated that the program 
anticipates more changes, including creation of a cargo requirement 
management board to decide the disposition of all change requests to 
production systems; establishment of a new invoice review policy; and 
colocation of personnel within a given business area. However, program 
officials told us that they have yet to define measures to determine the 
impact of such changes and thus are not yet positioned to determine their 
success. 

• Recommendation: Minimize the degree of overlap and concurrency 
across ongoing and future ACE releases and capture and mitigate the 
associated risks of any residual concurrence. 

In progress. Since May 2006, the program office has reduced overlap and 
concurrence of ACE releases and has taken actions to reduce potential 
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contention for limited resources by, for instance, decoupling (i.e., reducing 
dependencies among) certain program components; dividing releases into 
smaller subreleases to provide more flexibility in scheduling; improving 
planning for development, integration, testing, training activities, and 
milestones to better schedule use of development and test environments; 
and centralizing management of shared software services. Further, the 
program office conducts regular integration meetings with the teams 
supporting each release to discuss concerns, decisions, and schedules 
associated with resource availability and is using a software tool to track 
and mitigate release-specific concurrency risks. However, this tool 
contains vague or incomplete data relative to mitigating these risks. These 
data limitations make it difficult to determine the status or the 
effectiveness of the efforts to reduce the risks associated with overlap and 
concurrence among releases. 

• Recommendation: Direct the appropriate departmental officials to fully 
address those legislative conditions associated with having an approved 
privacy impact assessment (PIA) 5 and ensuring architectural alignment. 

In progress. One legislative condition states that the plan should meet 
OMB's capital planning and investment control review requirements, which 
include addressing security and privacy issues. The program office has 
developed a PIA for ACE release 4 (e-Manifest: Trucks), which is currently 
operational. DHS approved this PIA on July 14, 2006. This PIA addressed 
the major elements of DHS's guidance, and program officials stated that 
they would update the assessment for each ACE release. However, this PIA 
did not cover other recently completed screening releases. Program 
officials told us that these screening releases were considered to be part of 
the Automated Targeting System and were covered by the Automated 
Targeting System's PIA. However, this PIA does not specifically identify or 
address ACE screening releases. 

With respect to architectural alignment, DHS determined in May 2007 that 
all required ACE products and technologies were aligned with the DHS 
technical reference model and that ACE was thus aligned with the DHS 
enterprise architecture. However, we have yet to receive sufficient 
documentation describing the criteria and methodology used to make these 
determinations or verifiable analysis supporting the determinations. 
Moreover, the determinations were based on technical alignment and did 



5 44 U.S.C. § 3501 note. 
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not address other relevant aspects of program alignment to an enterprise 
architecture, such as data alignment. 

• Recommendation: Develop and implement key human capital 
management practices. 

In progress. In June 2006, the Office of Information Technology Strategic 
Human Capital Management Plan, which included an ACE-specific 
appendix as a road map for effective management of human capital 
resources, was approved. However, the plan does not address the basic 
tenets of effective human capital management, such as defining the 
positions needed (including core competencies) to perform core program 
functions, assessing and inventorying current workforce skills and abilities, 
assessing any gaps between needed and existing workforce levels and 
capabilities, and filling identified gaps. Officials acknowledged these 
limitations in the plans and stated that they are developing an 
implementation plan to address these shortfalls. 

• Recommendation: Include in the June 30, 2006, quarterly update report 
to the appropriations committees a strategy for managing ACE human 
capital needs and the ACE framework for managing performance and 
ensuring accountability. 

In progress. The June 30, 2006, quarterly report to the House and Senate 
Appropriations Committees included the ACE program's strategy for 
meeting its human capital needs and its accountability framework; 
however, the human capital strategy does not meet the basic tenets of 
strategic human capital management, as previously explained. 

• Recommendation: Accurately report to the appropriations committees 
on progress in implementing our prior recommendations. 

In progress. Quarterly reports to the House and Senate Appropriations 
Committees have contained information on the status of our open 
recommendations since November 2002, but recent reports remain 
outdated due to a 2- to 7-month time lapse between when the reports are 
produced and when they are provided to the appropriations committees. 
DHS and program officials stated that they are exploring ways to accelerate 
the review process and thereby improve the timeliness and accuracy of 
their reports to Congress. 
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Observations on the 
Expenditure Plan and 
Management of ACE 



We have three observations about ACE requirements, commercial product 
selection, and risk management. 

• Requirements: Redefinition of requirements for several ACE releases is 
now under way to address limitations in completeness of originally 
defined requirements, and this redefinition is likely to introduce 
program schedule delays and cost increases. 

In defining ACE requirements, the program office discovered that its 
original approach did not adequately engage all key stakeholders, such as 
software programmers and subject matter experts, for the legacy system 
ACE is intended to replace. To address this, key stakeholders are now 
collaborating and decomposing legacy code. However, this effort is 
expected to significantly delay some system releases and drops. Program 
officials have taken several actions that they say will minimize the impact 
of the delays, such as prioritizing shared functionality, dividing 
releases/drops into smaller increments, and changing release deployment 
strategies. However, these changes have yet to be approved, and the full 
extent of the cost and schedule implications is not yet known. Moreover, 
neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports 
have disclosed the requirements redefinition and its impact, and neither 
has addressed any changes to release deployment strategies. 

• Commercial product selection: Significant changes to ACE 
requirements have led to reevaluation and replacement of a key 
commercial off-the-shelf (COTS) product previously selected and being 
prepared for use. 

The program office conducted several analyses to determine which COTS 
products would best meet ACE system requirements, including a general 
review of various packages in 2002 to select a provider and a more detailed 
analysis in 2004 to define and allocate ACE requirements and select a 
specific product — the SAP Enterprise Portal. In December 2006, however, 
the ACE Chief System Architect determined through additional analysis 
that all planned SAP functionality could be provided by Internet 
Transaction Server (ITS) technology and recommended that ITS be 
adopted. The program office subsequently stopped work on SAP 
Enterprise Portal design and configuration efforts and reported that ITS 
would be used for release 5/drop A2 instead of the SAP Enterprise Portal. 
This decision is expected to have some near-term schedule impacts 
because much of the completed work for A2 had been based on the 
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planned use of SAP Enterprise Portal. Further, use of ITS raises the risk of 
inadequate user response time, which would, in turn, negatively impact 
user productivity and introduce a high probability of significant cost and 
schedule impacts. Program officials reported that actions are under way to 
mitigate the risk through performance modeling and test planning. 
However, neither the fiscal year 2007 expenditure plan nor the quarterly 
reports to Congress disclose this COTS product change, its impact on 
release schedules and cost estimates, or the risk to future system 
performance. 

• Risk management: All program risks are not being effectively managed. 

The program office has developed a process guide and implemented an 
automated tool (database) for managing ACE risks in accordance with 
relevant guidance and best practices. Although the database contains fields 
to provide a description, level (high, medium, or low), and mitigation 
strategy (including start and end dates, exit criteria, and implementation 
status) for each risk, the completeness and quality of this information 
varies. Because of such database limitations, we could not determine the 
status of and mitigation progress on 17 risks. Moreover, these database 
limitations were not reflected in the documentation used at key program 
events, indicating that the program does not have the risk-related 
information that it needs to inform its program decisions and to reduce the 
chances of potential problems becoming actual problems. Program 
officials stated that they are taking steps to improve risk management, 
including establishing a group to ensure the quality and completeness of 
the database, holding regular group meetings with contract staff and team 
leads to discuss risks and their impacts, and conducting risk management 
training. To date, however, program risks have not been communicated to 
oversight organizations through the 2007 expenditure plan or recent 
quarterly reports to the House and Senate Appropriations Committees. 



Conclusions Over the past 7 years, CBP and DHS have worked to fulfill legislatively 

mandated annual expenditure plan requirements and to implement dozens 
of our recommendations related to these plans and management of the 
program. Among other things, these requirements and recommendations 
have promoted effective program management and accountability for 
performance and results. As a result of these years of effort, the ACE 
program is better positioned today for delivering promised capabilities and 
benefits than it has been in the past. 
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Nevertheless, key program management practices relating to, for example, 
human capital management, requirements management, and risk 
management continue to remain a challenge, and other management areas, 
such as information security and architecture alignment, continue to 
require attention. As a result, avoiding major program schedule delays and 
cost overruns remains a challenge as more of each appears to be on the 
horizon. To further improve ACE management and minimize its exposure 
to risk, it is important for CBP and DHS to remain vigilant in their efforts to 
satisfy ACE legislative requirements and to fully implement our prior 
recommendations. Moreover, it is important that they keep Congress fully 
informed on where the program stands and what changes are planned to 
address emerging cost overruns and schedule delays. 



R6COmm6nd.cltiori.S for ^° further strengthen ACE management and promote accountability for 
-r-i ,. \ 4-' ACE performance and results, we are recommending that the Secretary of 

HiXtJCULlVt; jt\CL1UIL Homeland Security direct the CBP Commissioner to ensure that future 

quarterly reports to the House and Senate Appropriations Committees 

disclose 



• the risks and associated mitigation strategies of not having fully 
satisfied the expenditure plan legislative conditions and not having 
completed implementation of all our prior recommendations; 

• the status and impacts on the program's estimated cost and schedule 
and lessons learned from ongoing efforts to redefine requirements and 
to implement a different COTS product than originally selected; and 

• the program's plans and actions for improving ACE risk management 
and its current inventory of program risks, including their associated 
mitigation strategies and the status of the strategies' implementation. 



Agency Comments and 
Our Evaluation 



In written comments on a draft of this report signed by the Director, 
Departmental GAO/OIG Liaison, and reprinted in appendix II, DHS agreed 
with our findings and stated that it is committed to addressing them. 
Further, the department agreed with our recommendations and described 
actions that it said are under way or planned to address them. While most 
of the department's stated actions are consistent with our 
recommendations, in one case they may not be sufficient. Specifically, the 
department stated that it would ensure that future expenditure plans meet 
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all OMB capital planning and investment control review requirements by 
attaching the required OMB budget submission for ACE, referred to as an 
Exhibit 300, to all future plans. However, we reviewed the fiscal year 2007 
ACE Exhibit 300 as part of our review of the fiscal year 2007 ACE 
expenditure plan and found that it did not meet the OMB requirements 
cited above. Therefore, unless DHS improves the quality of future ACE 
Exhibit 300s by addressing the weaknesses we have identified, this action 
alone may not fully address our recommendation. 



We are sending copies of this report to the Chairmen and Ranking Minority 
Members of the other Senate and House committees and subcommittees 
that have authorization and oversight responsibilities for homeland 
security. We are also sending copies to the DHS Secretary, the CBP 
Commissioner and, on their request, to other interested parties. In addition, 
the report will be available at no charge on the GAO Web site at 
http://www.gao.gov. 

Should you or your offices have any questions on matters discussed in this 
report, please contact me at (202) 512-3459 or at hiter@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs may be 
found on the last page of this report. Other contacts and key contributors 
to this report are listed in appendix III. 




Randolph C. Hite 

Director, Information Technology Architecture 
and Systems Issues 
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Introduction 



The U.S. Customs and Border Protection 1 (CBP) is developing a new import and 
export processing system, referred to as the Automated Commercial Environment 
(ACE), to replace and supplement existing cargo processing technology. Begun in 
2001 , this system is being developed and deployed in a series of increments to 
about 300 ports of entry and is expected to be fully deployed by the end of 201 1 . 

The goals of ACE include 

• supporting border security by enhancing analysis and information sharing with 
other government agencies and providing CBP with the means to decide 
before a shipment reaches the border what should be targeted because it is a 
security threat and what should be expedited because it complies with U.S. 
laws; and 

• streamlining time-consuming and labor-intensive tasks for CBP personnel and 
the trade community through a national trade account and a single Web-based 
interface. 



The U.S. Customs and Border Protection (CBP), formerly the Bureau of Customs and Border Protection, was formed in 2003 under the 
new Department of Homeland Security from the former U.S. Customs Service and other entities with border protection responsibilities. 



3 
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Introduction 



The Department of Homeland Security Appropriations Act, 2007, 1 states that DHS 
may not obligate $216.8 million of the $316.8 million appropriated for ACE until the 
House and Senate Committees on Appropriations receive a plan for expenditure 
that 

1 . meets the capital planning and investment control review requirements 
established by the Office of Management and Budget (OMB), including 
Circular A-1 1 , part 7; 2 

2. complies with DHS's information systems enterprise architecture; 

3. complies with the acquisition rules, requirements, guidelines, and systems 
acquisition management practices of the federal government; 

4. includes a certification by the Chief Information Officer (CIO) of DHS that an 
independent verification and validation agent (IV&V) is currently under 
contract for the project; 



1 Pub. L. No. 109-295, (October 4, 2006). 

2 OMB Circular A-1 1 establishes policy for planning, budgeting, acquisition, and management of federal capital assets. 

4 
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Introduction 



5. is reviewed and approved by the DHS Investment Review Board (IRB), 1 the 
Secretary of Homeland Security, and OMB; and 

6. is reviewed by GAO. 

On February 6, 2007, DHS submitted its fiscal year 2007 expenditure plan for 
$316.8 million to the House and Senate Appropriations Subcommittees on 
Homeland Security. In addition, CBP submits quarterly reports to the House and 
Senate Appropriations Committees to keep them apprised of ACE progress and 
issues. 



1 The purpose of the IRB is to integrate capital planning and investment control, budgeting, acquisition, and management of 
investments. It is also to ensure that spending on investments directly supports and furthers the mission and that this 
spending provides optimal benefits and capabilities to stakeholders and customers. 

5 
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Objectives 



As agreed, our objectives were to 

1 . determine whether the ACE fiscal year 2007 expenditure plan satisfies the 
legislative conditions, 

2. determine the status of 15 open GAO recommendations for ACE, 1 and 

3. provide observations about the expenditure plan and DHS's management of 
the ACE program. 

We conducted our work at CBP headquarters and contractor facilities in the 
Washington, D.C., metropolitan area from December 2006 through July 2007 in 
accordance with generally accepted government auditing standards. Details of our 
scope and methodology are provided in attachment 1 . Related GAO products are 
in attachment 2. 



Two related open recommendations have been combined. 

6 
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Results in Brief: Objective 1 
Legislative Conditions 



Summary of satisfaction of legislative conditions 



Legislative condition 


Status 3 b 


1 . Meets the capital planning and investment control review requirements established by 
OMB, including OMB Circular A-1 1 , part 7. 


Partially satisfied 


2. Complies with the DHS enterprise architecture. 


Partially satisfied 


3. Complies with the acquisition rules, requirements, guidelines, and systems 
acquisition management practices of the federal government. 


Satisfied 


4. Includes a certification by the DHS CIO that an IV&V agent is currently under contract 
for the project. 


Satisfied 


5. Is reviewed and approved by the DHS IRB, Secretary of DHS, and OMB. 


Satisfied 


6. Is reviewed by GA0. 


Satisfied 



Source: GAO. 

"Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying 
many, but not all, key aspects of the condition that we reviewed. 

"Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every 
aspect of the condition that we reviewed. 
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Results in Brief: Objective 2 
Open Recommendations 



Implementation of prior GAO recommendations 3 



GAO Recommendation 


Status"'"-" 


1 . Define and implement an ACE accountability framework that ensures 




a. coverage of all program commitment areas, including key expected or estimated system (1) 
capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and 
schedules 


Complete 


b. currency, relevance, and completeness of such commitments made to Congress in 
expenditure plans 


In progress 


c. reliable data relevant to measuring progress against commitments 


Complete 


d. reporting in future expenditure plans progress against commitments contained in prior 
expenditure plans 


In progress 


e. use of criteria for exiting key readiness milestones that adequately consider indicators of 

system maturity, such as severity of open defects, and document milestone decisions in a way 
that reflects the risks associated with proceeding with unresolved severe defects and provides 
for mitigating these risks. e 


Complete 


2. Develop the range of realistic performance measures and targets needed to support an 
outcome-based, results-oriented accountability framework, including user satisfaction. 


Complete 


3. Explicitly align program goals, benefits, desired business outcomes, and performance 
measures. 


Complete 


4. Define measures and collect and use associated metrics for determining whether prior and 
future program management improvements are successful. 


Planned 
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Results in Brief: Objective 2 
Open Recommendations 



GAO Recommendation 


Status"*' -" 


5. Fully address those legislative conditions associated with measuring performance and results 
and employing effective IV&V practices. 


Complete 


3. Ensure that future expenditure plans are based on cost estimates that are reconciled with 
independent cost estimates. 


Complete 


7. Develop and implement a rigorous and analytically verifiable cost estimating program that 
embodies the tenets of effective estimating. 


Complete 


3. Use earned value management (EVM) f in developing all existing and future releases. 


Complete 


3. Have future expenditure plans specifically address any proposals or plans for extending and 
using ACE infrastructure to support other homeland security applications. 


Complete 


10. Minimize the degree of overlap and concurrence across ongoing and future releases, and 
capture and mitigate the associated risks of any residual concurrence. 


In progress 


1 1 . Fully address those legislative conditions associated with having an approved privacy impact 
assessment (PIA) and ensuring architectural alignment. 


In progress 


12. Develop and implement missing human capital management practices. 


In progress 


13. Include in the June 30, 2006, quarterly update report to the appropriations committees a 

strategy for managing human capital needs and the framework for managing performance and 
ensuring accountability. 


In progress 


14. Report to House and Senate Appropriations Committees on a quarterly basis on efforts to 
address open GAO recommendations. 


Complete 
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Results in Brief: Objective 2 
Open Recommendations 



GAO Recommendation 


Status abcd 


15. Accurately report to the appropriations committees on CBP's progress in implementing our prior 
recommendations. 


In progress 



Source: GAO. 

a With respect to the fiscal year 2007 expenditure plan. 

b Complete means that actions have been taken to fully implement the recommendation. 
c ln progress means that actions are under way to implement the recommendation. 
d Planned means actions are planned to implement the recommendation. 
This is a combination of two related recommendations. 

'EVM is a management tool for measuring progress and is both an industry accepted practice and an OMB requirement. 
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Results in Brief: Objective 3 
Observations 

Accountability * Integrity * Reliability 



Summary of observations 

• Limitations in the completeness of the original requirements for several 
ACE releases have resulted in changes to how requirements are being 
defined, as well as changes to the requirements themselves. These 
changes are likely to produce significant schedule delays and cost 
growth, and neither the expenditure plan nor the quarterly reports to 
Congress have disclosed these risks. 

• Requirements changes have led to reevaluation and replacement of a 
key commercial software product that was previously selected based on 
incomplete requirements. This change carries schedule and cost risks 
that neither the expenditure plan nor the quarterly reports have 
disclosed. 

• Management of ACE risks has not been effective, but improvements are 
planned to better anticipate and avoid problems that cause schedule 
delays and cost growth. ACE risks are not disclosed in either the 
expenditure plan or in the quarterly reports. 
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Results in Brief 
Recommendations and Agency Comments 



To reduce ACE exposure to future schedule delays and cost growth and to 
promote greater accountability, we are making recommendations to DHS aimed at 
disclosing the nature of and progress in addressing the risks associated with not 
having fully satisfied expenditure plan legislative conditions, not having completed 
implementation of all prior GAO recommendations, and having to redefine ACE 
requirements and reselect a key commercial software product. 

In oral comments on a draft of this briefing, DHS and CBP officials agreed with our 
conclusions and recommendations, and provided clarifying information and 
technical comments that we incorporated in the briefing, as appropriate. 
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Background 
Program Overview 



CBP is about 6 years into its trade processing modernization program, known as 
ACE. Among other things, ACE is to introduce reengineered business processes 
and next generation cargo processing technology, and it is to support CBP's 
mission of (1) protecting the American public against terrorism and (2) enforcing 
the laws of the United States while fostering our nation's economic security 
through lawful international trade and travel. 

ACE is also to support provisions of Title VI of the North American Free Trade 
Agreement, commonly known as the Customs Modernization Act. Subtitle B of 
the Act 1 contains provisions that were intended to enable the government to 
modernize international trade processes and permit CBP to adopt an informed 
compliance approach with industry using automated systems. 



1 19 U.S.C. Section 1411. 
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Background 
Program Overview 



The goals of ACE are to 

• enhance analysis and information sharing with other government agencies 
relative to new national security threats; 

• provide CBP personnel with the technology and information needed to decide, 
before a shipment reaches the border, what should be targeted because it is 
a security threat and what should be expedited because it complies with U.S. 
laws; 

• enable the efficient collection, processing, and analysis of commercial import 
and export data via an integrated, fully automated information system; 

• reduce costs for the government and the trade community by streamlining 
time-consuming and labor-intensive tasks; 

• enable government and trade communities to process, view, and manage 
accounts nationally and obtain historical information on cargo, conveyances, 
and crew, based on screening and targeting rules; and 

• enable government to comply with legislative mandates to improve 
efficiency/effectiveness and provide better customer service to U.S. citizens. 
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Several CBP components support the execution of the ACE program. 

• The Cargo Systems Program Office (referred to in this briefing as the program 
office) is responsible for the implementation of ACE. This office has primary 
responsibility for ACE program management; is responsible for the day-to-day 
management of ACE modernization activities; and develops the policies, 
standards, processes, and metrics by which the program is managed and 
measured. 

• The ACE program office is located within CBP's Office of Information 
Technology (OIT), which is headed by the Assistant Commissioner for 
Information and Technology. 

• The ACE program office is supported by the Targeting and Analysis Systems 
Program Office (a merger of ACE screening and targeting resources and the 
Office of Information Technology's Interprocess Solutions Branch), which is 
responsible for acquisition and development of ACE screening and targeting 
functionality. 

• The program office is supported by CBP's Office of Finance, Office of Strategic 
Trade, Office of Regulations and Rulings, and more than 100 participating 
government agencies and offices. 
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Background 
Contractors 

Accountability * Integrity * Reliability 

CBP awarded a 15-year, indefinite delivery/indefinite quantity, prime integration 
contract (5-year base with two 5-year options) to IBM Global Services in April 
2001 for development and implementation of ACE. CBP exercised the first option 
in April of 2006. 

• IBM and its subcontractors— collectively called the ACE Support 
Team 1 — provide ACE with program management support, enterprise 
engineering, systems planning and development, and systems business 
process reengineering; technology architectures; prototyping; systems and 
application design; software development, testing, and evaluation; 
deployment; and developmental operations support. 

• IBM is also under contract for operations and maintenance of deployed ACE 
releases, as well as for ongoing enhancements (new capabilities, referred to 
as program baseline enhancements) implemented in sub-releases that are 
scheduled in-between major ACE releases. 



1 This partnership was formerly known as the e-Customs partnership. It includes Lockheed Martin, Bearing Point, Sandler and Travis, Computer 
Sciences Corporation, and a number of small businesses. 
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CBP also relies on support contractors to provide a range of management 
support, such as program management, financial management, process 
improvement, quality assurance, requirements and configuration management, 
program communication, organizational learning, human capital planning, 
specialized cost analysis and life cycle cost estimating/modeling services, and 
procurement support. 

In 2002, ACE was to be completed in four increments over 4 years at an 
estimated cost of between $1 .5 billion and $1 .6 billion. 

The ACE Program Plan (version 2.1 , dated August 2006) states that ACE is 
expected to be fully deployed and operational by the end of 201 1 at a cost of 
about $3.3 billion. However, program officials told us that cost and schedule 
estimates are being revised, but have not yet been approved and therefore were 
not provided for our review. In the fiscal year 2007 ACE expenditure plan, CBP 
reports that it has spent about $1 .7 billion on the program. 

ACE is now being acquired and implemented through a series of 10 increments 
(referred to as releases), which are further divided into major system deliveries, 
known as "drops." (See the following six slides for a description of the increments 
and their status.) 
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A description of the 10 ACE increments (releases and related drops) follows. 

Release 1 (ACE Foundation): Computer hardware and system software 
(infrastructure) to support subsequent system releases. 

Release 2 (Account Creation): Initial group of national account managers 1 
and 41 importers access to account information, such as trade activity. 

Release 3 (Periodic Payment): Additional account managers and importers, 
as well as brokers and carriers, 2 with access to account information; 
provides initial financial transaction processing and revenue collection 
capability, allowing monthly payments of duties and fees. 

Release 4 (e-Manifest: Trucks): Electronic truck manifest 3 processing and 
interfacing to legacy enforcement systems and databases. 




GAP 
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1 CBP national account managers work with the largest importers to ensure their compliance with trade laws. 

2 Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a bond; carriers are 
individuals or organizations engaged in transporting goods for hire. 

3 Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane. 
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Release 5 (Entry Summary, Accounts, and Revenue (ESAR)): SAP 1 
technologies to enhance and expend accounts management, financial 
management, and entry summary functionality. 

• Master Data and Enhanced Accounts (drop A1): SAP to deliver 
enhanced account creation and maintenance functionality and expand the 
types of accounts managed in ACE. 

• Entry Summary and Revenue (drop A2): Entry summary, interfaces with 
participating government agencies, calculation of duties and fees, 
reconciliation processing, and refunds. 
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1 Systems Applications and Products (SAP) is a commercial enterprise resource planning software that has multiple modules, each performing 
separate but integrated business functions. ACE will use SAP to support many of its business processes and functions. In addition, the CBP's 
Modernization Office is using SAP as part of a joint project with its Office of Finance to support financial management, procurement, property 
management, cost accounting, and general ledger processes. 
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Release 6 (e-Manifest, All Modes, and Cargo Release): Electronic manifest 
capability for rail, air, and sea shipments; provides a multimodal manifest; 1 
enables full tracking of cargo, conveyances, individuals, and equipment; and 
enhances enforcement processes for rail, air, and sea. 

• e-Manifest: Rail and Sea Manifest (drop M1): Electronic manifest 
functionality for rail and sea shipments; rail, sea, and truck electronic 
manifests into the multimodal manifest. 

• e-Manifest: Air Manifest and Cargo Release (drop M2): Electronic 
manifest to air shipment and brings all modes of transportation into the 
multimodal manifest. 

• e-Manifest: Exports and Mail Entry Writing System (drop M3): 

Tracking of cargo, conveyances, individuals, and equipment for truck, 
sea, rail, and air manifests. 
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1 The multimodal manifest involves the processing and tracking of cargo as it transfers between different modes of transportation, for example, 
cargo arrives by ship, is transferred to a truck, and then is loaded onto an airplane. 
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Release 7 (Exports and Cargo Control): Remaining accounts management, 
revenue, manifest, release, and export functionality. 

• ESAR: Drawback, Protest, and Importer Activity Summary Statement 
(IASS) (drop A3): Import activity summary statement, 1 drawback 
functionality, and enhanced protest; online processing for trade account 
applications. 

• E-Manifest: Final Exports and Manifest (drop M4): Electronic manifest 
for mail, pipeline, and hand carry; electronic export processing. 
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1 An import activity summary statement is a summary of an importer's shipment activities over a specific period of time that is transmitted 
electronically to CBP on a periodic basis by importers and brokers. 
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Screening S1 (Screening Foundation): Foundation for screening cargo and 
conveyances by centralizing criteria and results into a single standard 
database; allows user definition and maintenance of data sources and 
business rules for air, rail, sea, and truck modes of transportation. 

Screening S2 (Targeting Foundation): Platform and foundation for advanced 
targeting capabilities by enabling CBP's National Targeting Center to search 
multiple databases for relevant facts and actionable intelligence and infer 
relationships between entities and data elements; architecture for integrating 
new data sources (including integrating external data sources and providing a 
single sign on capability), implementing analytical tools, and deploying 
analytical capabilities. 

Screening S3 (Advanced Targeting Capabilities): Screening for 

reconciliation, intermodal manifest, Food and Drug Administration data, and 
in-bond, warehouse, and foreign trade zone authorized movements; 
integrates additional data sources into targeting capability; and risk 
management capability. 
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According to CBP, as of July 2007, 

• Five increments are fully operational: Releases 1 , 2, and 3, and Screenings 
S1 and S2. 

• One increment (Release 4) has been deployed at 94 of the 99 truck land 
border ports. 

• Three increments are at various stages of development and/or deployment: 
Release 5/Drops A1 and A2, Release 6/Drop M1 , and Screening 3. 
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ACE Schedule 





2001 


2002 


2003 


2004 


2005 


2006 


2007 


2008 


2009 


2010 


2011 


Initial planning 


























Release 1 


























Release 2 


























Release 3 


























Release 4 


































Screening 1 


























Release 5 (A1) 
Release 5 (A2) 














































Screening 2 (TP) 
Screening 2 (TF) 


























Screening 3 


























Release 6 (M1) 
Release 6 (M2) 
Release 6 (M3) 
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Release 7 (A3) 
Release 7 (M4) 






















Planned schedule 

nrr7\ AHji ,s te H ^h»H„iP Today's briefing date: 7/26/07 


\ 



Source: GAO analysis based on CBP data. 
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Background 
Status of ACE Use 

Accountability * Integrity * Reliability 

CBP reports increased use of operational ACE increments. For example: 

• The number of external accounts 1 grew from 41 in June 2003 to 736 in 
August 2005, and stood at about 4,100 in October 2006. As of November 
2006, about 4,500 corporate entities had been approved to pay monthly 
duties and fees. 

• Total revenue collections through ACE grew from $84,673 in June 2004 to 
$1 .3 billion in July 2005, and as of October 2006 stood at about $8 billion. 

• The number of e-Manifests that were filed using ACE grew from 20,847 in 
December 2006 to 45,548 in January 2007. 



These accounts include importers, brokers, carriers, authorized service providers, and other members of the trade community who utilize ACE. 

25 



Page 35 



GAO-08-46 Customs Modernization 



Appendix I 

Briefing to Subcommittees on Homeland 
Security, House and Senate Committees on 
Appropriations 





Gr 


A f\ Background 
kJ Fundinq for ACE Expenditure Plans 


tHflfli^™ Accountability * Integrity * Reliability 












Budgeted amount 








Prior years 3 


Fiscal year 2007 


Total 






Release 












1 


ACE Foundation - through Operational Readiness 


114.8 




114.8 






2 


Account Creation - through Operational Readiness 








3 


Periodic Payment - through Operational Readiness 


172.9 




183.2 






4 


e-Manifest: Trucks - through Operational Readiness 


10.3 








5(A1) 


ESAR: Master Data and Enhanced Accounts 


172.3 




233.6 






5(A2) 


ESAR: Entry Summary and Revenue 


23.3 


38.0 






6 (M1) 


e-Manifest - Rail and Sea Manifest 


46.3 










6 (M2) 


e-Manifest - Air and Cargo Release 


10.4 


38.3 


95.0 






6 (M3) 


Exports and Mail Entry Writing System 












7 (A3) 


ESAR: Drawback, Protest and IASS b 












7 (M4) 


e-Manifest Custodial Entities, Pipelines, and Batch Processes 












S1 


Screening Foundation 


48.7 




48.7 






S2 


Targeting Foundation 


39.3 




39.3 






S3 


Advanced Targeting 


21.8 




21.8 






Activity 














ACE Operations and Maintenance 


143.1 


44.2 


187.3 








ACE Production Baseline Enhancements 




22.6 


22.6 








ACE Foundation Architecture and Engineering 


90.4 


35.7 


126.1 








ACE Implementation Infrastructure and Support 


269.5 


62.9 


332.4 








ACE Foundation Program Management 


137.5 


18.9 


156.4 








ACE Communications, Training, Outcome, and Deployment 


31.7 


24.4 


56.1 








Other Tasks 


20.6 




20.6 








Cargo Systems Program Office 


201.2 


25.3 


226.5 








International Trade Data System 


59.4 


16.0 


75.4 








Management Reserve 


92.0 


4.6 


96.6 








International Trade Data System Funding for Development 




•14.0 


-14.0 






Appropriated total 


1707.0 


317.0 


2024.0 








Source: GAO analysis of CBP data. 








a Prior funding consists of stopgap funding, approved by the appropriations committees in March 2001 , and the seven previous expenditure plans. 
b '° Funding for A3 and M4 has not yet been included in the expenditure plans. 

d This amount consists of prior year unobligated funds that ITDS provided to ACE to address requirements for participating government agencies. 
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Objective 1 : Legislative Conditions 



The 6 legislative conditions have been either satisfied or partially satisfied. 



Legislative condition 


Status 3 b 


1. Meets the capital planning and investment control review requirements established 
by the Office of Management and Budget (OMB), including OMB Circular A-1 1 , 
part 7. 


Partially satisfied 


2. Complies with DHS's enterprise architecture. 


Partially satisfied 


3. Complies with the acquisition rules, requirements, guidelines, and systems 
acquisition management practices of the federal government. 


Satisfied 


4. Includes a certification by the DHS CIO that an IV&V agent is currently under contract 
for the project. 


Satisfied 


5. Is reviewed and approved by the DHS IRB, Secretary of Homeland Security, and 
OMB. 


Satisfied 


6. Is reviewed by GAO. 


Satisfied 



Source: GAO. 



"Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for 
satisfying many, but not all, key aspects of the condition that we reviewed. 

"Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every 
aspect of the condition that we reviewed. 
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Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 



Condition 1. The plan, including related program documentation and program 
officials' statements, partially satisfies the capital planning and investment 
control review requirements established by OMB, including Circular A-1 1 , part 7. 1 

The table that follows provides an overview of the results of our analysis and 
selected examples in areas where A-1 1 requirements have or have not been fully 
satisfied. Given that the A-1 1 requirements are intended to minimize a program's 
exposure to risk, permit performance measurement and oversight, and promote 
accountability, any areas in which the program falls short of the requirements 
reduces the chances of delivering cost effective capabilities and measurable 
results on time and within budget. 



OMB Circular A-1 1 , part 7 establishes policy for planning, budgeting, acquisition, and management of federal capital assets. 
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Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 


Examples of A-11 Conditions Results of our analysis 






Provide a brief description of the 
investment and its status in the capital 
planning and investment control review 
process, including major assumptions 
made about the investment. 


The expenditure plan and the ACE Exhibit 300 budget submission provide brief 
descriptions of the ACE investment and its releases. The Exhibit 300 also 
describes the status of ACE relative to DHS's capital planning and investment 
control process, and states that ACE is in the "control" stage of the process. 

The Exhibit 300 and the ACE program plan also describe investment 
assumptions, such as (a) an annual budget of $305.5 million will be provided; 
(b) incremental deliveries within releases will mitigate risks and reduce 
uncertainty of program estimates; (c) ITDS funding will be timely, separate, and 
adequate; (d) stakeholder representation and resources will be adequate. 
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Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 




Examples of A-11 Conditions Results of our analysis 






Report performance goals and 
measures for existing investments and 
show how the Federal Enterprise 
Architecture (FEA) Performance 
Reference Model (PRM) applies to this 
investment. 


The expenditure plan reports the program's performance goals, benefits, 
objectives, performance measures, and their relationships for some, but not all, 
increments, including actual performance for fiscal years 2005 through 2006. 
The plan includes the ACE accountability framework, which is the program's 
overall tool for implementing CBP's PRM, which in turn is based on the FEA 
PRM. CBP's PRM includes, for example, performance measures for user 
satisfaction, efficiency, productivity, and data reliability, among other elements, 
and the ACE accountability framework incorporates a subset of these PRM 
measures, augmented by additional measures focusing on the program's 
status. 

ACE's Exhibit 300 also presents performance goals, targets, measures, and 
results for 2003 through 2012. However, these are not fully consistent with the 
expenditure plan. According to CBP, this is because the ACE performance 
measures were aligned with DHS's goals, objectives, strategies, and desired 
results as of July 2006, which is after the fiscal year 2007 Exhibit 300 was 
submitted. In addition, the performance measures continue to evolve as more 
understanding is gained in the usefulness and meaning of measures. (See the 
open recommendations section of this briefing for more information on 
performance measures.) 
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A G AO 

Accountability * Integrity * Reliability 


Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 




Examples of A-11 Conditions Results of our analysis 






Provide a summary of the investment's 
risk assessment, including how 19 OMB- 
identified risk elements are being 
addressed. 


The expenditure plan presents some, but not all, of the identified risks for the 
program's releases and activities. The Exhibit 300 also presents program risks, 
assessments, mitigation strategies, and status, and it organizes them by OMB 
risk categories. However, the risks in the expenditure plan and the Exhibit 300 
are not fully consistent with each other or with the risks being tracked in the 
program's risk database. Further, the most recent risk assessment is limited to 
security risks for Release 4. 

The ACE program has a documented risk management process that includes 
identifying, classifying, reporting, and tracking risks. However, this process has 
not been fully implemented. For example, some risks that are identified in the 
risk database and the accountability framework have missing or outdated status 
information and mitigation strategies. As a result, the status of all risks is not 
clear. Program officials stated that risk management is not yet mature, but that 
they are taking steps to improve it. (See the open recommendation section of 
this briefing for more information on risk management.) 
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A G AO 

Accountability * Integrity * Reliability 


Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 




Examples of A-11 Conditions Results of our analysis 






Provide a summary of the investment's 
status in accomplishing baseline cost and 
schedule goals through the use of an 
earned value management (EVM) 
system or operational analysis, 
depending on the life-cycle stage 


The expenditure plan reported some, but not all, EVM data for ACE releases 
and activities. For example, the data includes planned, estimated, and actual 
schedule milestones, but it does not include schedule variances. 

The EVM data in the expenditure plan was from the ACE accountability 
framework that program officials rely on to manage the program. (See the open 
recommendations section of this briefing for more information on the 
accountability framework.) According to program officials, they are using EVM 
to manage all ACE releases and screenings under contract and EVM tracking 
begins with an Integrated Baseline Review, which forms the basis for a realistic 
plan against which to objectively measure work to be completed during the 
period of performance. 

The Exhibit 300 also includes EVM data and describes contractor requirements 
for EVM, including verification of EVM compliance with industry standards. 
According to program officials, these contractual requirements are still 
operative. 

Nevertheless, the EVM program has still not been certified, as required by 
OMB. According to program officials, this certification is scheduled for the 
beginning of fiscal year 2008. As an interim measure, the prime contractor 
performed an EVM compliance and internal surveillance audit against relevant 
EVM criteria 1 in March 2006 and found no issues. 






1 EVM is a management tool for measuring progress and is both an industry accepted practice and an OMB requirement. 
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Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 


Examples of A-11 Conditions Results of our analysis 






Provide a description of the investment's 
privacy and security issues. Summarize 
the agency's ability to manage security at 
the system or application level. 
Demonstrate compliance with the 
certification and accreditation processes 
as well as the mitigation of IT security 
weaknesses. 


Privacy. The expenditure plan does not discuss privacy issues. However, the 
Exhibit 300 states that ACE is subject to the privacy provisions of the E- 
Government Act of 2002. It further states that a privacy impact assessment was 
conducted in 2005 for Release 4 and that separate assessments will be 
conducted for each subsequent release. This assessment was approved by 
DHS on July 14, 2006, and our analysis shows that it complied with relevant 
DHS guidance. This assessment does not cover other recently completed 
screening releases S1 and S2. According to program officials, S1 and S2 are 
considered to be part of the Automated Targeting System (ATS), and are 
therefore covered by the ATS PIA. However, our analysis of the ATS PIA 
showed that while it addresses screening and targeting functions, it does not 
specifically identify or address releases S1 and S2. (See the open 
recommendations section of this briefing for further information on privacy 
issues.) 

Security. CBP has taken a number of actions related to ACE security 
management. It conducted a security self-assessment for Release 4 in 
September 2006 using the National Institute of Standards and Technology 
(NIST) Security Self-Assessment Guide for Information Technology Systems 
and did not report any major security vulnerabilities. In addition, CBP officials 
stated that another security self-assessment was conducted in April 2007. 
However, we have yet to receive this self-assessment. CBP also reports that it 
has conducted two full contingency plan tests at its disaster recovery facility. 
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Objective 1 : Legislative Conditions 
Condition 1 : Partially Satisfied 



Examples of A-11 Conditions Results of our analysis 



Privacy and Security (cont.) 



Further, CBP accredited ACE Release 4 on November 22, 2004; Release S1on 
July 18, 2006; and Release S2 on October 18, 2006. The program office plans to 
conduct its next ACE certification and accreditation of ACE in September 2007 
for Release 5/Drop A1 . However, since November 2004, the deployed ACE 
system has been subject to considerable change. Specifically, about 7,100 
trouble tickets, trouble reports, change requests and install requests were 
generated — some of which have resulted in system changes. Federal guidance 
recommends reaccreditation whenever significant changes to the system or its 
operational environment are likely to affect a system's security posture. 1 
However, CBP has not established explicit criteria for when to 
reaccredit/recertify a system in the face of changes, or documented a systematic 
procedure for determining the security risk from the cumulative impact of 
changes made to the system. As a result, CBP has not recertified or 
reaccredited the system. 

In addition, the ACE security plan and security risk assessment have not been 
updated to reflect key information, such as the results of the April 2007 security 
self-assessment or a system access control risk identified in May 2006 and 
included in the program's risk database. 



Source: OMB and NIST criteria and GAO analysis of DHS documentation. 



1 Guide for the Security Certification and Accreditation of Federal Information Systems, NIST Special Publication 800-37. NIST, Gaithersburg, MD, May 2004. 
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Objective 1 : Legislative Conditions 
Condition 2: Partially Satisfied 



Condition 2. The plan, including related program documentation and program 
officials' statements, partially satisfies the condition that it comply with the DHS 
information systems enterprise architecture (EA) as currently defined. 

According to federal guidelines and best practices, investment compliance with 
an EA is essential for ensuring that an organization's investment in new and 
existing systems is defined, designed, and implemented in a way that promotes 
integration and interoperability and minimizes overlap and redundancy, thus 
optimizing enterprisewide efficiency and effectiveness. A compliance 
determination is not a one-time event that occurs when an investment begins, 
but is, rather, a series of determinations that occurs throughout an investment's 
life cycle as changes to key aspects of both the EA and the investment's 
architecture are made (e.g., data, business, services, and technology). 

The DHS Enterprise Architecture Board, supported by the Enterprise 
Architecture Center of Excellence, is responsible for ensuring that projects 
demonstrate adequate technical and strategic compliance with the department's 
EA. 
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Objective 1 : Legislative Conditions 
Condition 2: Partially Satisfied 



During 2006, the DHS Enterprise Architecture Board conducted three reviews of 
ACE architectural alignment with the DHS EA Technical Reference Model (TRM) 1 
as part of key milestone reviews for three release/screening increments. 

• Master Data and Enhanced Accounts (Drop A1), Critical Design Review, 
June 2006; 

• E-Manifest: Rail and Sea (Drop M1), Critical Design Review, September 
2006, and 

• Targeting Foundation (S2), Production Readiness Review, November 2006. 

On May 1 , 2007, DHS EA officials reported that all required products and 
technologies were aligned to the TRM, and thus that ACE was in alignment with 
the DHS EA. In addition, the DHS CIO certified as part of the above milestone 
reviews that each increment was in alignment with the TRM. More specifically, 

• In December 2006, the Center of Excellence determined that ACE was 
conditionally compliant with the DHS EA, contingent on actions taken in 
regard to four conditions. Three of the four conditions were resolved by 
March 2007. 

1 A technical reference model is list of approved IT industry standards, hardware, and software products that ensures that IT solutions developed by 
individual programs (such as ACE) are consistent within DHS as a whole. 
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Objective 1 : Legislative Conditions 
Condition 2: Partially Satisfied 



• On March 29, 2007, the EAB recommended approval of ACE's decision 
request for program alignment, with one remaining condition — that by June 
30, 2007, the program submit technology insertions packages for the 
products identified as needing alignment with the DHS TRM. 

Notwithstanding these EA compliance determinations, DHS did not provide us 
with sufficient documentation of its determinations to allow us to understand the 
methodology and criteria, or to verify the analyses, that were used to arrive at 
them. Moreover, documentation that was provided showed that the 
determinations focused on ACE technical alignment, and did not address, for 
example, alignment to the DHS EA Data Reference Model. 

Until DHS demonstrates, through verifiable documentation and methodologically- 
based analysis, that ACE is aligned with all relevant aspects of the DHS EA, 
including corporate data structures and standards, the program will be at risk of 
being designed and implemented in a way that does not support optimized 
departmental operations, performance, and achievement of strategic goals and 
outcomes, including those related to information sharing. 
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Objective 1 : Legislative Conditions 
Condition 3: Satisfied 

Accountability * Integrity * Reliability 

Condition 3. The plan, including related program documentation and program 
officials' statements, satisfies the condition that it comply with the acquisition 
rules, requirements, guidelines, and systems acquisition management practices 
of the federal government. 1 

Federal acquisition rules, requirements, guidelines, and management practices 
provide an acquisition management framework that is based on the use of 
rigorous and disciplined processes for planning, managing, and controlling the 
acquisition of IT resources. 2 These acquisition management processes are 
embodied in published best practices models, such as the Capability Maturity 
Models® developed by Carnegie Mellon University's Software Engineering 
Institute (SEI). These models explicitly define, among other things, acquisition 
process management controls that are recognized hallmarks of successful 
organizations and that, if implemented effectively, can greatly increase the 
chances of acquiring software-intensive systems that provide promised 
capabilities on time and within budget. 



1 For this condition, we did not evaluate the compliance of the ACE program with respect to the Federal Acquisition Regulation, OMB directives, or 
other governmentwide requirements. 

2 See, for example, the Clinger-Cohen Act of 1996 (P.L No. 104-106, 40 U.S.C. §§11101 through §§11704) and OMB Circular A-1 30. 
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Objective 1 : Legislative Conditions 
Condition 3: Satisfied 

Accountability * Integrity * Reliability 

In our prior reviews of the ACE program, 1 we reported that ACE had 
satisfied this condition based on SEI's November 2003 assessment of the 
program against the Software Acquisition Capability Maturity Model (SA- 
CMM®). 2 That assessment assigned the program an SEI level 2 3 rating, 
indicating that CBP had instituted basic acquisition management 
processes and practices. Since receiving its 2003 rating, the program 
office has not conducted another acquisition capability assessment to 
ensure that it is continuing to employ these basic acquisition controls, and 
does not plan to do so. 

Furthermore, program officials told us that although the prime contractor 
was contractually required to have an SEI CMM® Level 3 4 capability, they 
are no longer required to do so because funding contractor program 
management-related activities is now a lower priority relative to other 
competing demands for ACE funding. 

1 GAO-06-580; GAO-05-267. As with our prior reviews of ACE expenditure plans, the evaluation does not include compliance with federal acquisition 
regulations or other federal rules and requirements beyond those encompassed by SEI's Capability Maturity Models. 

2 The SA-CMM® is consistent with the acquisition guidelines and systems acquisition management practices of the federal government, and provides a 
management framework that defines acquisition practices for such process areas as acquisition planning, solicitation, requirements development and 
management, project management, contract tracking and oversight, and evaluation. 

3 ACE's level 2 rating indicated that CBP had instituted basic acquisition management processes and practices consistent with the acquisition 
guidelines and management practices of the federal government in the following areas: acquisition planning, solicitation, requirements development 
and management, project management, contract tracking and oversight, and evaluation. 

4 Level 3 capability is more advanced than level 2 and indicates that acquisition management processes have been defined throughout the organization. 
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Objective 1 : Legislative Conditions 
Condition 3: Satisfied 

Accountability * Integrity * Reliability 

According to CBP officials, several steps have been taken to mitigate any impact 
of no longer focusing on SEI CMM® compliance, such as 

• requiring the prime contractor and the program office's support contractors to 
follow plans, processes, and procedures for such key areas as EVM, 
configuration management, and problem reporting; 

• having the program office's quality assurance staff, in collaboration with the 
prime contractor's quality assurance function, monitor adherence to key 
management controls through reviews of ACE processes and products, 
including 48 such reviews in fiscal year 2006; and 

• having the program office's contracting function continuously monitor 
adherence to contractual provisions; 

• leveraging OIT assistance, through the Process Asset Group, to conducts 
reviews of new and updated plans, policies, and procedures; 1 and 

• cataloguing job aids in a Process Asset Group library that specifies the 
required activities for program management processes and disseminating 
them to ACE staff. 

1 The program office refers to these plans, processes, and procedures as assets. 
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Objective 1 : Legislative Conditions 
Condition 4: Satisfied 

Accountability * Integrity * Reliability 

Condition 4. The plan satisfies the condition that it include certification by the 
DHS CIO that an IV&V agent is currently under contract. 

On October 24, 2006, the DHS Deputy CIO certified in writing that an IV&V agent 
is under contract for ACE and that the agent met applicable requirements and 
standards. (See open recommendations section of this briefing for more 
information on IV&V.) 
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Objective 1 : Legislative Conditions 
Condition 5: Satisfied 

Accountability * Integrity * Reliability 

Condition 5. The plan, including related program documentation and program 
officials' statements, satisfies the requirement that it be reviewed and approved by 
the DHS IRB, the Secretary of Homeland Security, and OMB. 

• The DHS IRB reviewed the program and approved the expenditure plan on 
December 12, 2006. 

• The DHS Under Secretary for Management approved the expenditure plan on 
behalf of the Secretary of Homeland Security on February 6, 2007. 

• OMB approved the expenditure plan on January 22, 2007. 
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Objective 1 : Legislative Conditions 
Condition 6: Satisfied 

Accountability * Integrity * Reliability 

Condition 6. The plan satisfies the requirement that it be reviewed by GAO. 
Our review was completed on July 26, 2007. 
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Objective 2: Open Recommendations 
Accountability Framework 



Open Recommendation 1: Define and implement an ACE accountability 
framework that fulfills these conditions: 

a. Covers all program commitment areas, including key expected or estimated 
system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; 
and (4) milestones and schedules. 

Status: Complete 

Effective program management includes defining and measuring progress against 
program commitments and being held accountable for results. Such commitments 
generally cover expected or estimated (1) capabilities and their associated use 
and quality, (2) benefits and mission value, (3) costs, and (4) milestones and 
schedules. 

Since 2003, we have reported that such commitments for ACE have not always 
been defined, although improvements have been made. Most recently, we 
reported 1 that the program office had developed an initial version of an 
accountability framework for measuring several program commitments, such as 
capabilities and milestones, but that other commitments, such as benefits, had 
not been as well defined. 



1 GAO-06-580. 
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Objective 2: Open Recommendations 
Accountability Framework 



During the last year, the program office has continued to improve its coverage of 
program commitments, as evidenced by the content of key program documents — 
ACE accountability framework, ACE Program Plan (August 2006), the CBP/ACE 
performance reference model, periodic Program Management Review (PMR) 
reports, and the fiscal year 2007 expenditure plan. For example, 

• The accountability framework now captures and integrates data on ACE 
functional and performance capabilities, benefits, and mission value, 
estimated and actual costs, milestones and schedule, accomplishments, and 
earned value management status. Further, the framework provides visibility 
into each ACE release at several levels of detail, and it is being used by the 
ACE Executive Director and others as the means for monitoring program 
progress, issues, and decisions. 

• The PMR reports have included the accountability framework data. 

• The fiscal year 2007 ACE expenditure plan included an example of the 
accountability framework commitments from the September 2006 PMR 
report. 
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Objective 2: Open Recommendations 
Accountability Framework 



b. Ensures currency, relevance, and completeness of such commitments made to 
Congress in expenditure plans. 

Status: In progress 

We have previously reported that commitments made in expenditure plans relative 
to program capabilities, benefits, costs, and schedules need to be current, 
relevant, and complete. To the extent that they are not, the currency and 
relevance of the plan and its utility to Congress as an accountability mechanism 
are limited. 

ACE expenditure plans have not always included such information. To address 
this limitation, we reported last year 1 that CBP was relying on quarterly reports to 
Congress to provide the appropriations committees with more current, relevant, 
and complete information about the program than could be provided in the 
expenditure plan. However, we also noted that the quarterly reports were generally 
submitted to Congress 3 to 4 months after the end of each quarter, thus limiting 
their currency and relevance. 



1 GAO-06-580. 
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Objective 2: Open Recommendations 
Accountability Framework 



The fiscal year 2007 expenditure plan continues to include information about 
program commitments that is not current. Specifically, 

• The latest expenditure plan was provided to the appropriations committees 
on February 6, 2007. However, information from the ACE accountability 
framework that was included in the plan, such as milestones, EVM values, 
and risks, were as of October 2006, making it about 4 months old. For 
example, the expenditure plan listed the milestone for the Production 
Readiness Review (PRR) for a key release, Release 5/Drop A1 , as March 
1 , 2007. However, this milestone had already slipped by more than 4 
months--to July 12, 2007--by the December 2006 PMR (held on January 5, 
2007). 
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Objective 2: Open Recommendations 
Accountability Framework 



According to program officials, they continue to use the quarterly ACE status 
reports to provide the appropriations committees with more current information. 1 
However, recent quarterly reports have not been current. For example, 

• The December 2006 quarterly report was not provided until February 26, 
2007, and it contained the same outdated PRR milestone as the 
expenditure plan did for the release previously mentioned. 

• Other quarterly reports have been submitted to the appropriations 
committees as many as 7 months after the end of the quarter. 

CBP and DHS officials told us that the delays were due to the DHS review and 
approval process and that incorporating the most current program information 
would delay the reports even longer. According to the ACE and DHS officials, 
they are exploring ways to accelerate the review process. 



1 Quarterly reports should be received by Congress approximately 60 days after the end of the quarter. 
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Objective 2: Open Recommendations 
Accountability Framework 



c. Ensures reliable data relevant to measuring progress against commitments. 
Status: Complete 

The quality of the capabilities that a program is to deliver is a relevant program 
commitment. One measure of the quality of system capabilities is the trend in the 
number and severity of unresolved system defects or problems. Reliable data 
about these defects are needed so that system maturity can be understood and 
informed investment decisions can be made. 

We previously reported 1 that ACE defect data were not always consistent 
because the two tools that the program used to track defects were not integrated 
and reconciled. As a result, the true status of ACE defects, and thus an important 
measure of system quality, was not known. 

Since then, the program office has integrated the two tools using a cross- 
referencing function and has instituted manual processes for reconciling the data 
in each tool. The December 2006 quarterly congressional report stated that these 
efforts have improved the ability to record, assess, and report on system quality 
and performance. 

1 GAO-06-580. 
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Objective 2: Open Recommendations 
Accountability Framework 



d. Ensures reporting in future expenditure plans progress against commitments 
contained in prior expenditure plans. 

Status: In progress 

The value and utility of an expenditure plan for congressional oversight and 
agency accountability depends in large part on whether the plan reports on 
progress against commitments-system capabilities, benefits, costs, and 
schedules-made in prior plans. 

Historically, the ACE expenditure plans have not reported adequately on 
progress against previous plan commitments. Most recently, we reported 1 that 
the fiscal year 2006 expenditure plan contained several such reporting gaps, 
such as whether funding amounts were actually obligated and expended as 
planned and whether releases met schedules. In particular, we reported that the 
plan did not address whether the design and development for Release 5 was 
actually accomplished as planned. 



1 GAO-06-580. 
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Objective 2: Open Recommendations 
Accountability Framework 



The fiscal year 2007 expenditure plan still does not adequately describe the 
program's progress against the commitments that were made in the fiscal year 
2006 plan. For example, 

• The plan does not report progress against milestones in the fiscal year 
2006 plan or explain why these milestones were not achieved. 

• The plan does not report actual obligation or expenditure of funds relative 
to the planned uses of these funds in prior expenditure plans. 

• The plan did not address progress against the milestone dates for each 
stage of a release that was included in the prior year's plan. As a case in 
point, the fiscal year 2006 expenditure plan described specific functionality 
planned for Release 5/Drop A1 (Master Data and Enhanced Accounts), 
such as online registration for trade representatives. However, the fiscal 
year 2007 expenditure plan does not include information on whether these 
functions were delivered, stating only that Release 5 functionality would be 
deployed in two phases. 
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Objective 2: Open Recommendations 
Accountability Framework 



According to program officials, the quarterly congressional reports provide more 
current information on the program's progress against prior commitments. 
However, we found that these reports have also not fully addressed the 
commitments made in prior expenditure plans. For example, recent quarterly 
reports describe progress against developmental milestones for each release, but 
do not report whether key functionality associated with each release was 
delivered. In particular, the report for the first quarter of fiscal year 2007 identified 
planned future capabilities and associated milestones for Release 5/Drop A1 but 
did not address the status of the key functions associated with this release. 

Program officials stated that they intend to start providing this information in the 
last quarterly report of each year. 
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Objective 2: Open Recommendations 
Accountability Framework 



e. Ensures use of criteria for exiting key readiness milestones that adequately 
consider indicators of system maturity, such as severity of open defects, and 
document the milestone decisions in a way that reflects the associated risks and 
plans for mitigating them. 1 

Status: Complete 

As noted earlier, one measure of the quality of system capabilities being 
delivered is the number and severity of unresolved system defects or problems. 
As such, information on such defects is an important consideration when 
program decisions, such as key milestone decisions, are made. 

We previously reported 2 that several key milestones were passed that had 
severe open defects and that program officials were unable to provide 
documentation regarding how the risks associated with these defects were 
assessed. We also reported that these risks were not being tracked in the 
program's risk database. 

1 This is a combination of two prior recommendations. From GAO-05-267, "Define and implement an ACE accountability framework that ...ensures 
use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects" and from 
GAO-06-580, "Document key milestone decisions in a way that reflects the risks associated with proceeding with unresolved severe defects and 
provides for mitigating these risks." 

2 GAO-06-580. 
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Objective 2: Open Recommendations 
Accountability Framework 



Since then, the system life cycle gate review process for Office of Information 
and Technology projects, including ACE, has been amended to include 
milestone readiness decisions based on an assessment and acceptance of risks 
(including the risks related to unresolved defects). Further, the ACE Risk and 
Issue Management Process guidance provides for the systematic identification, 
analysis, prioritization, planning, execution, evaluation, and documentation of 
program risks and issues and requires that any risks associated with going 
forward should be identified as part of each life cycle gate review. 

This process has been applied in the following two major gate reviews: 

• Release A1 (ESAR) Critical Design Review (May 2006) and 

• Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review 
(August 2006). 
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Objective 2: Open Recommendations 
Accountability Framework 



In the case of Release 5/Drop A1 , program officials reported that no severe 
defects existed, but that all known risks associated with proceeding past the 
milestone had been entered into the program's risk database and documented in 
the certification package submitted to the DHS CIO. 

The quarterly reports to Congress similarly state that risks related to gate review 
decisions and the associated impacts are entered into a database to ensure 
visibility and mitigation and are included in the package submitted to the DHS CIO 
for review and certification to pass a given milestone. 

Going forward, officials stated that similar assessments will be part of the 
following planned gate reviews for Release 5/Drop A1 : 

• Operational Readiness Review (planned for August 23, 2007) and 

• Live operations (planned for August 25, 2007). 
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Objective 2: Open Recommendations 
Performance Measures 

Accountability * Integrity * Reliability 

Open recommendation 2: Develop the range of realistic ACE performance 
measures and targets needed to support an outcome-based, results-oriented 
accountability framework, including user satisfaction with ACE. 

Status: Complete 

We have previously reported on both the absence of meaningful performance 
measures to understand ACE progress, quality, and results, and have raised 
concerns about the practicality and applicability of some measures that have been 
defined. Most recently, we reported 1 that defined ACE performance targets are not 
always realistic and that goals, expected mission benefits, and performance 
measures are not fully defined and adequately aligned. 

In response, the program office has established an initial set of performance 
measures that are tied to program objectives and related performance goals, and it 
has established processes for the collecting, analyzing, and integrating 
performance data for each measure. Among other things, these performance 
measures address user satisfaction, efficiency, and productivity. Moreover, these 
measures have been made an integral part of the ACE accountability framework 
and they were approved by the CBP Commissioner and the DHS CIO on June 30 
and July 6, 2006, respectively. 

1 GAO-06-580. 
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Objective 2: Open Recommendations 
Performance Measures 

Accountability * Integrity * Reliability 

According to program officials, these initial measures will continue to be revised 
and augmented, and thus will evolve over time based on lessons learned, changes 
to existing release capabilities, and refinements of requirements for future releases. 
For example, the program office reported in March 2007 that new measures are 
still being developed, reviewed, and evaluated for Releases 2, 3, and 4 at the same 
time that performance data is being collected and reported for existing measures 
for these releases. According to program officials, the new measures will be 
combined with existing performance measures. 

In addition, CBP's December 2006 quarterly congressional report stated that 
performance measures for future releases, such as Release 5, are being 
scheduled for identification and development and that changes are being made to 
existing measures when they are determined to be inappropriate. For example, the 
program office learned that the performance measure "percentage of truck 
manifests being filed electronically" did not recognize that empty trucks are not 
required to file manifests, and thus it revised the metric used to divide the number 
of electronically filed truck manifests by the total number of required truck manifests 
instead of the total number of trucks in order to get a more meaningful reflection of 
ACE performance. 

To assist in managing these performance measures, the program has established 
a life cycle process for performance measures as well as a database tool to record 
and manage modifications. 
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Objective 2: Open Recommendations 
Performance Measures Alignment 



Open recommendation 3: Explicitly align ACE program goals, benefits, desired 
business outcomes, and performance measures. 

Status: Complete 

As just discussed, the program office has developed an initial set of ACE program 
measures that are tied to program goals, among other things, and they intend to 
continue to update and evolve these as the program moves forward. In addition, 
the program office has developed an ACE Performance Reference Model that 
explicitly aligns the CBP strategic goals, objectives, strategies, and desired results 
applicable to the ACE program with specific performance measures. For example, 



CBP Strategic Goal 


CBP Objective 


CBP Strategy 


Desired Result 


Performance Measure 


Preventing terrorism at ports of 
entry: Prevent terrorists and 
terrorist weapons, including 
weapons of mass destruction 
and weapons of mass effect, 
from entering the U. S. 


Improve information 
and targeting 


Use advanced passenger and 
cargo information (NTC, ATS-Air, 
ATS, Screening and Targeting- 
ACE) to pre-screen, target, and 
identify potential terrorists and 
terrorist shipments and any related 
activity. 


Increased use of 
targeting. 


Number of security-focused 
selections generated by 
system (= or above intensive 
threshold) by type 



Further, this model links the measures to specific ACE releases. For example, the 
above performance measure applies to Screening Foundation (S1) and Targeting 
Foundation (S2). 
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Objective 2: Open Recommendations 
Management Improvement Measures 



Open recommendation 4: Define measures and collect and use associated 
metrics for determining whether prior and future program management 
improvements are successful. 

Status: Planned 

As we have previously reported, 1 investments in program management 
improvements should include defined measures of progress and results. To date, 
the program office has implemented a number of such improvements; however, it 
has not had measures or metrics to determine the success of the improvements. 
Moreover, the December 2006 quarterly congressional report stated that the 
program anticipates more changes, including 

• creation of a cargo requirement management board to decide the disposition 
of all change requests to production systems; 

• establishment of a new invoice review policy; and 

• co-location of personnel within a given business area. 



1 GAO-04-719. 
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Objective 2: Open Recommendations 
Management Improvement Measures 



According to its December 2006 quarterly reports to Congress, the program office 
plans to measure the impact of future management improvements. However, 
program officials told us that they have yet to define them and thus are not yet 
using such measures. 



60 



Page 70 



GAO-08-46 Customs Modernization 



Appendix I 

Briefing to Subcommittees on Homeland 
Security, House and Senate Committees on 
Appropriations 



k G AP 

nH^^™ Accountability * Integrity * Reliability 



Objective 2: Open Recommendations 
Legislative Conditions 



Open recommendation 5: Fully address those legislative conditions associated 
with measuring ACE performance and results and employing effective IV&V 
practices. 

Status: Complete 

Among the legislative conditions that ACE expenditure plans have been required to 
meet are satisfaction of OMB guidance, including that associated with measuring 
program performance and results and use of effective IV&V practices. These 
conditions reflect good program management practices and, if implemented 
properly, can reduce program risks. 

Performance and Results 

As previously discussed in this briefing, the program office has developed a range 
of ACE performance measures and taken steps to align them with program, CBP, 
and DHS strategic goals and outcomes. According to program officials, they plan to 
continue to evolve and refine the performance measures and include the measures 
in the ACE accountability framework. 
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Objective 2: Open Recommendations 
Legislative Conditions 



IV&V Practices 

In January 2006, the DHS CIO certified that an IV&V agent was under contract for 
the ACE program, but noted several issues, including these: 

• mechanisms were needed to ensure that products were complete, of sufficient 
quality, and met the needs of the user and 

• a more explicit technical approach, describing when and how certain activities 
should (or should not be) performed, was needed. 

Moreover, we subsequently reported 1 that the scope of the contractor's activities 
did not extend to both IV&V of key system products and development processes. 



1 GAO-06-580. 
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Objective 2: Open Recommendations 
Legislative Conditions 



On October 24, 2006, the Deputy CIO again certified that an IV&V agent was under 
contract and stated that the previous issues had been addressed. Since then, the 
program office has also developed an IV&V Implementation Management Plan that 
addresses the concerns raised by the DHS CIO and us. In particular, the plan 

• requires an IV&V program consistent with the industry standard; 

• provides a set of objectives, guidelines, and expectations for IV&V activities, 
including periodic independent reports on status, observations, 
recommendations, and activities; and 

• addresses satisfaction of quality standards for ACE products and user needs. 

Program officials told us that IV&V has allowed early identification and correction of 
program process and product weakness. 
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Objective 2: Open Recommendations 
Reconciliation of Cost Estimates 

Accountability * Integrity * Reliability 

Open recommendation 6: Ensure that future expenditure plans are based on 
cost estimates that are reconciled with independent cost estimates. 



Status: Complete 

It is important that expenditure plans be based on reliable estimates of costs, to 
include reconciling differences between government and independent cost 
estimates. We recently reported 1 that the cost estimate in the fiscal year 2006 
expenditure plan was based on government and independent cost estimates that 
had been compared and found to be consistent. 

For the fiscal year 2007 expenditure plan, our analysis showed that the 
government and independent cost estimates differed by about 15 percent. 
According to program officials, they reconciled these differences in January 2007, 
and concluded that the results did not warrant changes to the expenditure plan 
because the government estimates used in the expenditure plan were more 
accurate. 



1 GAO-06-580. 
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Objective 2: Open Recommendations 
Reconciliation of Cost Estimates 

Accountability * Integrity * Reliability 

According to program officials, two primary factors account for the difference in 
estimates. Specifically, 

• The estimates assumed different timelines for completing development of all 
releases. The independent estimate assumed development completion by 
fiscal year 2010, while the government estimate assumed fiscal year 201 1 . 
According to program office officials, this accounts for about five percent of the 
difference. 

• The government estimate included a number of items that the independent 
estimator did not. For example, the independent estimator did not include 
training and outreach items, such as the cost of conferences with trade 
associations, training materials, and associated travel. As a result, the 
government estimated training and outreach costs over the life of the program 
to be about $90 million, while the independent estimate put these costs at 
about $39 million. The independent estimate has since been amended to 
include the missing items, and the reconciliation process is continuing. 
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Objective 2: Open Recommendations 
Rigorous Cost Estimation 



Open recommendation 7: Develop and implement a rigorous and analytically 
verifiable cost estimating program that embodies the tenets of effective estimating. 1 

Status: Complete 

The reliability of cost estimates is largely a function of the quality of the estimating 
process used to derive them. We previously reported 2 that the program did not 
have a well-defined cost estimating process, but that it has since made progress in 
strengthening its cost estimating program. Specifically, the program office has: 

• defined and documented processes for estimating program costs (including 
management reserve costs) and 

• hired a contractor to develop costs estimates that were independent of the 
government estimates. 

In September 2006, an ACE support contractor reported that both the government 
and independent cost estimation processes demonstrated significant conformance 
to effective estimating practices and concluded that the program is using a rigorous 
and verifiable cost estimating approach. 

1 See, for example, models developed by the Carnegie Mellon University SEI, Checklists and Criteria for Evaluating the Cost and Schedule Estimating 
Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University, 1995) and A Manager's Checklist for 
Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa.: Carnegie Mellon University, 1995). 

2 GAO-04-719, GAO-05-267, GAO-06-580. 
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Objective 2: Open Recommendations 
Earned Value Management 



Open recommendation 8: Use EVM in developing all existing and future 
releases. 

Status: Complete 

EVM is a program management tool to measure progress by comparing, during a 
given period of time, the value of work accomplished with the amount of work 
expected to be accomplished. This comparison permits performance to be 
evaluated based on calculated variances from the planned (baselined) cost and 
schedule. EVM is both an industry accepted practice and an OMB requirement. 

We recently reported 1 that the program office was not using EVM for all releases 
(e.g., Release 5) due to changes to release baselines and the lack of familiarity 
on the part of program staff with EVM practices. 

Since then, the program office has established the performance baselines needed 
to implement EVM for Release 5/Drops A1 and A2, Screening 3, and Release 
6/Drop M1. 



1 GAO-06-580. 
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Objective 2: Open Recommendations 
Earned Value Management 



For these releases, the program office reports that it has also applied EVM 
standards in establishing baselines and included EVM data in the accountability 
framework for management and decision-making purposes. Program officials also 
reported that they plan to implement EVM on future releases and task orders. 
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Objective 2: Open Recommendations 

ACE Infrastructure 

Accountability * Integrity * Reliability 

Open recommendation 9: Have future ACE expenditure plans specifically 
address any proposals or plans, whether tentative or approved, for extending and 
using ACE infrastructure to support other homeland security applications, 
including any impact on ACE of such proposals and plans. 

Status: Complete 

Together, the fiscal year 2007 expenditure plan and the quarterly reports to 
Congress address ACE's relationships with other trade processing and DHS 
applications. For example, the plan discusses efforts to work with participating 
government agencies in defining and deploying the International Trade Data 
System (ITDS). The stated goal is to deliver ACE/ITDS in an integrated and 
coordinated manner. In this regard, the plan discusses workshops to gather 
requirements from participating agencies for Release 6/Drop M2 (first held on July 
19, 2006), a working group to address these agencies' HAZMAT issues 
(established on July 17, 2006), and efforts to develop data element inputs for 
ACE from other government agencies. 
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Objective 2: Open Recommendations 

ACE Infrastructure 

Accountability * Integrity * Reliability 

In addition, the quarterly reports to Congress cite coordination efforts with related 
homeland security programs. For example, the report for the first quarter of 2007 
states that 

• Container Security Initiative will be supported by ACE Release 6 and 
Screening 1-3 capabilities; 

• Customs-Trade Partnership Against Terrorism is coordinating with ACE 
Release 5/Drop A1 capabilities to provide both CBP and trade 
representatives with the ability to view the status of CBP programs; and 

• U.S. -Mexico Border Partnership Plan will coordinate with ACE to implement 
any cargo screening standards derived from partnership plan agreements. 

In addition, CBP reports that ACE and other CBP applications 1 share 
infrastructure (desktops, clients, and local area networks) at the ports of entry. 
This infrastructure is purchased and maintained by CBP's OIT Program 
Integration Division. Each application, including ACE, is responsible for complying 
with OIT's infrastructure standards. 



iCBP applications include IT systems that provide tools and information to help front-line officers ensure the security of our nation. This includes 
applications for the United States Visitor and Immigrant Status Indicator Technology (US-VISIT), Container Security Initiative and Automated 
Targeting System. 
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Objective 2: Open Recommendations 
Overlap and Concurrence 



Open recommendation 10: Minimize the degree of overlap and concurrency 
across ongoing and future ACE releases and capture and mitigate the associated 
risks of any residual concurrence. 

Status: In progress 

Significant overlap and concurrency among major program activities, such as 
releases, introduce considerable cost and schedule risks as they can create 
contention for limited resources among the releases. We have continued to report 
on extensive overlap and concurrence in ACE releases and the cost overruns and 
schedule delays that have resulted. Most recently, we reported 1 that the ACE 
schedule continued to provide for such overlap and concurrency and that the risks 
associated with doing so were not being effectively addressed. 

Since then, the program office has reduced this overlap and concurrence. For 
example, it has 

• decoupled (i.e., reduced dependencies among) certain ACE program 
components by separating Screening 1-3 from Releases 4-6 and 

• aligned the development and delivery of functionality for different releases 
with the availability of required hardware environments. 

1 GAO-06-580. 
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Objective 2: Open Recommendations 
Overlap and Concurrence 



Recent quarterly congressional reports stated that CBP has taken actions to 
reduce potential contention for limited resources. Examples include 

• dividing releases into smaller subreleases, called drops, to provide more 
flexibility in scheduling; 

• improving planning for development, integration, testing, and training activities 
and milestones to better schedule use of development and test environments; 
and 

• centralizing management of shared software services to address, among 
other things, allocation of resources and responsiveness to workload peaks 
and use of consistent technical management approaches across releases. 

Moreover, the program's risk database identifies overlap and concurrency-related 
risks. However, the mitigation strategies for these risks contain vague or 
incomplete data. These data limitations make it difficult to determine the status or 
the effectiveness of the efforts to reduce the risks associated with overlap and 
concurrence among releases. (See the observations section of this briefing for 
additional information on program risk management.) 
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Objective 2: Open Recommendations 
Overlap and Concurrence 



The program office is also conducting regular integration meetings with the teams 
supporting each release to discuss concerns, decisions, and schedules associated 
with resource availability and is using a software tool to track and mitigate release- 
specific concurrency risks. 

Notwithstanding these steps, the program continues to face challenges in 
managing dependencies among releases, which any associated concurrency in the 
program's development exacerbates. For example, the schedule slips expected 
with Release 5/Drop A2 will affect Release 6/Drop M1 , as resources have been 
shifted from M1 to address the A2 delay. In addition, further delays in A1 and A2 
will impact the implementation of M1 . 
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Objective 2: Open Recommendations 
Legislative Conditions 



Open recommendation 11 : Direct the appropriate departmental officials to fully 
address those legislative conditions associated with having an approved privacy 
impact assessment (PIA) 1 and ensuring architectural alignment. 

Status: In progress 

The department approved the ACE PIA for Release 4 in July 2006, however, this 
assessment does not cover other completed releases. Further, DHS has 
determined that ACE is aligned with DHS architecture, but we have yet to receive 
documentation to adequately understand and verify the determination. 

PIA (In progress) 

In March 2006, DHS developed guidance on the development and content of a 
PIA. As we have previously reported, 2 the purpose of a PIA is to ensure that there 
is no collection, storage, access, use, or dissemination of identifiable personal or 
business information that is not both needed and permitted. Development and 
use of a PIA are both a requirement of OMB and the E-Government Act of 2002. 3 

1 44 U.S.C. § 3501 note. 
2 GAO-06-580. 

3 OMB, Guidance for Implementing the Privacy Provisions of the EGovernment Act of 2002, OMB M-03-22 (Sept. 26, 2003). 
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Objective 2: Open Recommendations 
Legislative Conditions 



The program office has developed a PIA for Release 4 of the ACE e-Manifest: 
Trucks and the International Trade Data System, and DHS approved it on July 14, 
2006. As noted earlier, our analysis shows that this PIA addresses the major 
elements of DHS's guidance. Program officials stated that they would update the 
assessment for each ACE release. This assessment does not cover other 
recently completed screening releases (i.e., S1 and S2). However, program 
officials told us that S1 and S2 are considered to be part of the Automated 
Targeting System (ATS), and are therefore covered by the ATS PIA. However, 
our analysis of the ATS PIA showed that while it addresses screening and 
targeting functions, it does not specifically identify or address releases S1 and S2. 

Architectural Alignment (in progress) 

As discussed in the legislative conditions section of this briefing, DHS has 
determined that ACE is aligned with the DHS EA. For example, 

• In December 2006, DHS's EA Center of Excellence determined that ACE 
was conditionally compliant with the DHS architecture, but that the program 
needed to take actions to address four conditions. 
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Objective 2: Open Recommendations 
Privacy Assessment and Architectural Alignment 



• On March 2007, the DHS EA Board reported that ACE had satisfied three of 
the conditions and recommended approval of ACE's request for program 
alignment with one remaining condition — alignment with the DHS Technical 
Reference Model. 

• In May 2007, DHS EA officials reported that all required products and 
technologies were aligned with the Technical Reference Model and that ACE 
was thus aligned with the DHS EA. 

However, as was also discussed earlier in this briefing, we have yet to receive 
sufficient documentation describing the criteria and methodology used to make 
these determinations or verifiable analysis supporting the determinations. 
Moreover, the determinations were based on technical alignment and did not 
address other relevant aspects of program alignment to an EA, such as data 
alignment. Thus, the program is at risk of being designed and implemented in a 
way that is not consistent with all relevant aspects of the DHS EA, such as data 
structures and standards, and thus does not support optimized DHS-wide 
operations, performance, and results. 
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Objective 2: Open Recommendations 
Human Capital Practices 



Open recommendation 12: Develop and implement key human capital 
management practices. 

Status: In progress 

As we have previously reported, 1 effective strategic management of program 
human capital includes, among other things, defining the skill sets needed to 
perform program functions, assessing and inventorying the skill sets of the 
program's current workforce and assessing any associated skill gaps in meeting 
future needs, and developing strategies to fill any identified gaps. Moreover, it 
includes having a well-defined plan that provides for effective implementation of 
these processes. 

In June 2006, CBP executives approved the OIT Strategic Human Capital 
Management Plan. The plan is intended to be a road map for supporting CBP's 
workforce vision and ensuring effective management of human capital resources 
across OIT to address enterprise-wide priorities. An ACE-specific plan was 
included as an appendix to the CBP plan. This appendix was intended to provide 
better near- and long-term human capital management practices for the OIT 
offices involved in ACE development. 

1 GAO-06-580. n 
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Objective 2: Open Recommendations 
Human Capital Practices 



Neither the OIT nor the ACE-specific plan addresses the basic tenets of effective 
human capital management. For example, neither provides for: 

• defining the positions needed (including core competencies) to perform core 
program functions; 

• assessing and inventorying current workforce skills and abilities; 

• assessing any gaps between needed and existing workforce levels and 
capabilities; and 

• filling identified gaps via such means as hiring new staff, training existing 
staff, and augmenting staff with contractor support. 

OIT officials acknowledged these limitations in the plans and stated that they are 
developing an implementation plan to address these shortfalls. The officials 
stated that the implementation plan will include accountability, timeframes, and 
metrics to carry out the larger OIT Strategic Human Capital Management Plan. 

According to these officials, this implementation plan was to be completed in 
January 2007, following presentation to the OIT Deputy Director's Council and 
approval by the OIT Office Directors and Assistant Commissioner. However, as of 
July 2007, it had not yet been approved and thus was not available for our review. 
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Objective 2: Open Recommendations 
Human Capital Practices 



In the interim, OIT and program officials reported taking various steps to 
address ACE human capital needs. These include, for example, 

• Using various methods to fill staff vacancies. As of May 2007, the program 
reports that it has 62 full-time employees and one vacancy and that 
additional staff are working on ACE either full- or part-time. 

• Lobbying OPM for more flexibility in recruiting, to include higher salaries 
and direct hiring authority. 

• Reorganizing OIT into six program offices aligned to major mission areas 
so that the number of government personnel responsible for ACE 
development activities can be augmented by IT functional program 
management expertise. 

• Offering training to improve the skills of staff. 

Nevertheless, these steps have not been guided by a well-defined plan and 
thus represent activities that are not tied to strategic goals and outcomes and 
thus cannot be measured against them. Without a plan, it is unlikely that the 
program will be able to adequately ensure that it has the right people at the 
right time to deliver ACE successfully. 
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Objective 2: Open Recommendations 

Quarterly Reports 



Open recommendation 13: Include in the June 30, 2006, quarterly update report 
to the appropriations committees a strategy for managing ACE human capital 
needs and the ACE framework for managing performance and ensuring 
accountability. 

Status: In progress 

The June 30, 2006 quarterly report to the House and Senate Appropriations 
Committees included the program's strategy for meeting its human capital needs 
and its accountability framework. 

Human Capital Strategy (In progress) 

The June 30, 2006, quarterly report to Congress included a description of the 
previously described ACE-specific appendix to the OIT Strategic Human Capital 
Management Plan, including the plan's five goals and strategies. Moreover, the 
report states that this ACE-specific plan is aligned with OPM's Human Capital 
Assessment and Accountability Framework. 1 As previously stated, however, this 
ACE-specific plan does not meet the basic tenets of strategic human capital 
management, as defined in this framework and other relevant guidance. 

1 As revised by OPM in 2005, this framework reflects guidance from the collaboration of OMB, OPM, and GAO. 
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Objective 2: Open Recommendations 

Quarterly Reports 



According to program officials, they are developing a new OIT human capital 
strategy and implementation plan, but they have yet to provide us with a date when 
it will be completed. 

Accountability Framework (Complete) 

The June 30, 2006, quarterly report included a description of the ACE 
accountability framework. As previously discussed, this framework is the means by 
which the program measures performance relative to promised ACE capabilities, 
costs, schedules, earned value, risks, and mission values and benefits. The report 
also included an appendix illustrating the format and content of the accountability 
framework tool. At that time, CBP reported that it would continue to work with 
stakeholders to further enhance the format, readability, and utility of the framework 
as a program management and reporting tool. 

Since June 2006, the program office has refined and implemented the 
accountability framework. The framework covers all program commitment areas 
and key system aspects to support executive decision making and provides 
external program stakeholders with information on the program. 
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Objective 2: Open Recommendations 
Quarterly Report on Open Recommendations 



Open recommendation 14: Report quarterly to the House and Senate 
Appropriations Committees on efforts to address open GAO recommendations. 

Status: Complete 

CBP has submitted quarterly reports on ACE to both the House and the Senate 
Appropriations Subcommittees since November 2002, including reports for each 
quarter of fiscal year 2006 and the first and second quarters of fiscal year 2007. 
These reports have addressed CBP's efforts to address open GAO 
recommendations. 
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Objective 2: Open Recommendations 
Accurately Report on Open Recommendations 



Open recommendation 15: Accurately report to the appropriations committees 
on CBP's progress in implementing our prior recommendations. 

Status: In progress 

As previously stated, CBP has included information on progress in meeting 
GAO's recommendations in its quarterly reports 1 to the appropriations committees 
since November 2002. However, some of this information is dated and thus 
inaccurate due to the time lapse between when the reports are produced and 
when they are provided to the appropriations committees. Recently, this time 
lapse has been between about two and seven months. According to program 
officials, they are exploring ways to accelerate the review process and thereby 
improve the timeliness and accuracy of their reports to Congress. DHS officials 
also told us that they have ongoing efforts to improve the review process; 
however, the process has not improved consistently or significantly. 



1 The reports are due to Congress approximately 60 days after the completion of a fiscal quarter. ACE quarterly reports are due to DHS approximately 
30 days after the end of the quarter, and then DHS has 30 days to review the report and submit it to Congress. 
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Objective 3: Observations 
Requirements Limitations Likely to Cause Major Delays 



Observation 1: Redefinition of requirements for several ACE releases is now 
under way to address limitations in completeness of originally defined 
requirements, and this redefinition is likely to introduce program schedule delays 
and cost increases. 

A key aspect of successful system acquisition programs is having well-defined 
requirements. Among other things, requirements should be complete, and to 
accomplish this, best practices advocate engaging all key stakeholders in the 
requirements definition and management process. 

In defining ACE requirements, the program office discovered that its original 
requirements definition approach did not adequately engage all key stakeholders, 
and to its credit, has since taken steps to address this. Specifically, 

• In the spring of 2004, the program office and the ACE support team conducted 
more than 300 business process workshops with the ACE user community to 
help define ACE requirements for the future releases. This requirements 
definition approach was referred to as the Global Business Blueprint (GBB) 
effort. 
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Objective 3: Observations 
Requirements Limitations Likely to Cause Delays 



In June and July of 2005, a key group of ACE stakeholders that were not 
originally involved in the GBB (application programmers familiar with the 
legacy system that ACE is to replace) raised questions about the 
completeness of the requirements. To address these questions, the program 
office examined the GBB's completeness by first reverse-engineering the 
legacy software for a small number of legacy system programs and then 
comparing the reverse-engineered requirements to the GBB-derived 
requirements. Based on this, which is referred to as legacy code 
decomposition, the program office found that the GBB-derived requirements 
were missing about 20 percent of needed ACE functionality. 

In August 2005, the program office decided that it needed to decompose all of 
the legacy system code in order to completely capture the requirements for all 
ACE releases. Work began with decomposition of the legacy code related to 
ACE Release 5/Drop A2 (Entry Summary Accounts and Revenue). 
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Objective 3: Observations 
Requirements Limitations Likely to Cause Delays 



In September 2006, the program office determined that the legacy code 
decomposition approach alone was not sufficient to gain a full understanding 
of the requirements given the size and complexity of A2. As a result, the 
requirements definition process was expanded to engage another key 
stakeholder group (business process subject matter experts). Under the 
expanded approach, referred to as legacy code decomposition and 
collaboration, legacy system software programmers and subject matter 
experts were to examine the code line by line in defining ACE requirements. 

In November 2006, the legacy system code decomposition and collaboration 
effort for Release 5/Drop A2 fell behind schedule because of lack of personnel 
with legacy system expertise and experience. The schedule for A2 was 
tentatively revised, but at the time of our review, the schedule was still under 
review, had not yet been approved, and thus was not yet available for our 
review. 
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Objective 3: Observations 
Requirements Limitations Likely to Cause Delays 



The program office has identified the A2 legacy code decomposition and 
collaboration process as a high risk item that will significantly impact the A2 
schedule. For example, the decomposition and collaboration process for part of the 
functionality on A2's critical path — the Authorized Broker Interface Entry 
Summary — is not expected to be completed before early 2008, at the earliest. 

According to program officials, delays for other releases/drops, such as Release 
5/Drop A1 and Release 6/Drop M1 , will not be as significant. They also said that, 
while they have not yet estimated how long A2 will be delayed and what the 
associated cost implications are, they do not expect the cost increases to breach 
the current acquisition program baseline of $3.3 billion, which translates into a cost 
increase of less than $200 million. Moreover, they said that several actions have 
been taken to minimize the impact of the delays. For example, 

• A2 functionality necessary for M1 has been given priority in order to support 
M1 deployment as originally planned and 

• A2 scope is being divided into increments to allow some functionality to be 
delivered sooner and to minimize the impact on other drops. 
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Objective 3: Observations 
Requirements Limitations Likely to Cause Delays 



However, these actions carry consequences, such as missed opportunities to 
combine field training and an increase in the number of legacy interfaces, thus 
increasing the potential for introducing software problems. 

In addition, program officials told us that they are considering changes to the A2 
and M1 deployment strategies to address stakeholder concerns, and they said that 
these changes could also minimize the magnitude of the A2 and M1 delays. 
Specifically, they said that they had planned to deploy on a national basis, meaning 
that A2 and M1 functionality would be deployed to all ports at the same time and 
concurrently adopted by all users nationwide. However, deployment may change to 
a filer-by-filer basis, meaning that A2 and M1 functionality would be deployed to all 
ports at the same time, but not all filers would begin using it at the same time. 
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Objective 3: Observations 
Requirements Limitations Likely to Cause Delays 



According to program officials, they believe that the change in the deployment 
strategy would both address stakeholder concerns and minimize any A2 and M1 
schedule delays caused by the redefinition of requirements. However, they added 
that these changes have yet to be approved, and the full extent of the cost and 
schedule implications are not yet known. Moreover, neither the fiscal year 2007 
expenditure plan nor the ACE quarterly reports have disclosed the A2 and M1 
requirements redefinition and its impact, and neither has addressed any changes to 
their deployment strategies. 
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Objective 3: Observations 
Key COTS Product Being Replaced 



Observation 2: Significant changes to ACE requirements have, in turn, led to 
reevaluation and replacement of a key commercial off-the-shelf product (COTS) 
previously selected and being prepared for use. 

When acquiring commercial component-based systems, like ACE, best practices 
advocate basing decisions on whether to employ a given COTS product on 
thorough, rigorous, and continuous analysis of a number of factors, including how 
well competing products satisfy defined system requirements. 

To the program office's credit, it reports having followed the CBP system life cycle 
methodology to determine which COTS product would best meet the requirements 
for Release 5/Drop A2. These analyses include: 

• In 2002, the program office reviewed, in general terms, various COTS 
packages and determined that a solution using SAP (formerly Systems 
Application and Products), a COTS provider, combined with other commercial 
solutions and customized development, provided the best combination of 
capability, performance, and cost for ACE. 
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Objective 3: Observations 
Key COTS Product Being Replaced 



In 2004, a more detailed analysis was conducted as part of the previously 
mentioned GBB process, which was intended to define and allocate ACE 
requirements to future ACE releases and provide the basis for, among other 
things, selecting a specific SAP product. At that time, the SAP Enterprise 
Portal product was selected for Release 5/Drop A2. 

In December 2006, the ACE Chief System Architect recommended that the 
Internet Transaction Server (ITS) technology already used by CBP should be 
adopted instead of the SAP Enterprise Portal, based on the determination that 
all currently planned SAP functionality could be presented using the ITS 
technology. This decision was based on improved understanding of the 
requirements and previous analyses of ITS. 
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Objective 3: Observations 
Key COTS Product Being Replaced 



On the basis of these analyses and the Chief System Architect's recommendation, 
the program office subsequently stopped work on SAP Enterprise Portal design 
and configuration efforts and, in March 2007, the program reported that the SAP 
Internet Transaction Server would be used for Release 5/Drop A2 instead of the 
SAP Enterprise Portal. While this decision was expected to have some near-term 
schedule impact because much of the completed work for A2 had been based on 
the planned use of SAP Enterprise Portal, the program office reports that these 
impacts are offset by the cost advantages of other ACE releases already using the 
Internet Transaction Server technology. 

However, in March 2007, program officials identified a risk of inadequate response 
time for the Internet Transaction Server — thus negatively impacting user 
productivity — and that there was high probability of significant cost and schedule 
impacts. Actions are underway to mitigate the risk through performance modeling 
and test planning. 

Neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress 
disclose this COTS product change, its impact on release schedules and cost 
estimates, or risk to future system performance. 
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Objective 3: Observations 
Risks Not Being Effectively Managed 



Observation 3: All program risks are not being effectively managed. 

Risk management is a continuous, forward-looking process that is intended to 
either prevent program cost, schedule, and performance problems from occurring 
or to minimize the impact if they occur. According to relevant guidance and best 
practices, effective risk management involves proactively identifying, assessing, 
and disclosing risks; defining and implementing cost-effective strategies for 
mitigating these risks; and measuring and disclosing progress in doing so. 

To its credit, the program office has developed a process guide and implemented 
an automated tool (database) for managing ACE risks in accordance with relevant 
guidance and best practices. Among other things, the database contains the 
description, level (high, medium, or low), and mitigation strategy (including start 
and end dates, exit criteria, and implementation status) for each risk. 
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Objective 3: Observations 
Risks Not Being Effectively Managed 



However, the completeness and quality of the information on each of the risks in 
the risk database 1 vary. For example, 

• many risks were missing information on the status of efforts to implement the 
mitigation strategy and 

• many risks (18) were missing criteria for completing mitigation steps, clear 
descriptions of what the risk entailed, and start and end dates for planned 
mitigation activities. 

Because of such database limitations, we could not determine the status of and 
mitigation progress on 17 risks. Moreover, these database limitations were 
reflected in the documentation used at key program events, such as PMRs. This 
means that the program does not have the risk-related information that it needs to 
inform its program decisions and to reduce the chances of potential problems 
becoming actual problems. 



1 As of May 23, 2007 the risk database contained 46 risks. 
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Objective 3: Observations 
Risks Not Being Effectively Managed 



Program officials, including the official responsible for risk management, stated that 
risk management is immature and needs to be strengthened. The reasons they 
gave for these risk management weaknesses are due to 

• all staff not being trained on how to use the tool (last training was provided in 
2003, and the since then a number of people have joined the program); 

• the tool is unique to the ACE program and thus no CBP guidance exists on its 
use); and 

• each ACE group addresses risk differently in its weekly meetings. 
To improve ACE risk management, program officials told us that they are: 

• establishing a group to ensure the quality and completeness of the database; 

• holding regular group meetings with contract staff and team leads to discuss 
risks and their impacts; and 

• conducting risk management training. 
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Objective 3: Observations 
Risks Not Being Effectively Managed 



If implemented effectively, such steps should result in more meaningful information 
about program risks that can be useful to DHS in managing the program and to 
Congress in overseeing it. To date, however, ACE program risks have not been 
communicated to oversight organizations through the 2007 expenditure plan or 
recent quarterly reports to the House and Senate Appropriations Committees. 
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Conclusions 



Over the past 7 years, CBP and DHS have worked to fulfill legislatively mandated 
annual expenditure plan requirements and to implement dozens of our 
recommendations related to these plans and management of the program. Among 
other things, these requirements and recommendations have promoted effective 
program management and accountability for performance and results. As a result 
of these years of efforts, the ACE program is better positioned today for delivering 
promised capabilities and benefits than it has been in the past. 

Nevertheless, key program management practices relating to, for example, human 
capital management, requirements management, and risk management continue to 
remain a challenge, and other management areas, such as information 
security and architecture alignment, continue to require attention. As a result, 
avoiding major program schedule delays and cost overruns remains a challenge as 
more of each appear to be on the horizon. 
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Conclusions 



To further improve ACE management and minimize its exposure to risk, it is 
important for CBP and DHS to remain vigilant in their efforts to satisfy ACE 
legislative requirements and to fully implement our prior recommendations. 
Moreover, it is important that they keep the Congress fully informed on where the 
program stands and what changes are planned to address emerging cost overruns 
and schedule delays. 
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Recommendations for Executive Action 



To further strengthen ACE management and promote accountability for ACE 
performance and results, we are making the following recommendation to the 
Secretary of Homeland Security to direct the CBP Commissioner to ensure that 
future quarterly reports to the House and Senate Appropriations Committees 
disclose 

(1) the risks and associated mitigation strategies of not having fully satisfied the 
expenditure plan legislative conditions and not having completed 
implementation of all our prior recommendations; 

(2) the status and impacts on the program's estimated cost and schedule and 
lessons learned from ongoing efforts to redefine requirements and to 
implement a different COTS product than originally selected; and 

(3) the program's plans and actions for improving ACE risk management and its 
current inventory of program risks, including their associated mitigation 
strategies and the status of the strategies' implementation. 
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Agency Comments 



In oral comments on a draft of this briefing, DHS and CBP officials agreed with our 
conclusions and recommendations, and provided clarifying information and 
technical comments that we incorporated in the briefing, as appropriate. 
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Attachment 1 
Scope and Methodology 



Scope and Methodology 

To accomplish our objectives, we analyzed the ACE fiscal year 2007 expenditure 
plan and supporting documentation, and compared them to relevant federal 
requirements and guidance, applicable best practices, and our prior 
recommendations. We also interviewed DHS and CBP officials and ACE program 
contractors. In particular, we reviewed 

• DHS and CBP investment management practices, using OMB A-1 1 , part 7; 

• DHS and CBP certification activities for ensuring ACE compliance with the 
DHS enterprise architecture; 

• DHS and CBP acquisition management efforts, using SEI's SA-CMM; 

• CBP cost estimating program and cost estimates, using SEI's institutional and 
project-specific estimating guidelines; 1 



1 SEI's institutional and project-specific estimating guidelines are defined in Robert E. Park, Checklists and Criteria for 
Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: 
Carnegie Mellon University Software Engineering Institute, 1995) and A Manager's Checklist for Validating Software Cost 
and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburg, Pa: 1995), respectively. 
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Attachment 1 
Scope and Methodology 



independent verification and validation (IV&V) activities using the Institute of 
Electrical and Electronics Engineers standard for software verification and 
validation; 2 

CBP actions to coordinate ACE with related programs; 

CBP's reorganization documentation, including the organizational charts and 
roles and responsibilities matrix; 

ACE's accountability framework; and 

cost and schedule data and program commitments from program 
management documentation. 



institute of Electrical and Electronics Engineers (IEEE) Computer Society, Standard for Software Verification and 
Validation 1012-1998 (June 8, 2005). 



102 



Page 112 



GAO-08-46 Customs Modernization 



Appendix I 

Briefing to Subcommittees on Homeland 
Security, House and Senate Committees on 
Appropriations 



k G AP 

nH^^™ Accountability * Integrity * Reliability 



Attachment 1 
Scope and Methodology 



For DHS-, CBP-, and contractor-provided data that we did not substantiate, we 
have made appropriate attribution indicating the data's source. 

We conducted our work at CBP headquarters and contractor facilities in the 
Washington, D.C., metropolitan area from December 2006 through July 2007 in 
accordance with generally accepted government auditing standards. 
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Attachment 2 
Related GAO Products 



ACE Expenditure Plans 

Information Technology: Customs Has Made Progress on Automated 
Commercial Environment System, but IT Faces Long-Standing Management 
Challenges and New Risks. GAO-06-580. Washington, D.C.: May 31 , 2006. 

Information Technology: Customs Automated Commercial Environment 
Program Progressing, but Need for Management Improvements Continues. 
GAO-05-267. Washington, D.C.: March 14, 2005. 

Information Technology: Early Releases of Customs Trade System Operating, 
but Pattern of Cost and Schedule Problems Needs to Be Addressed. G AO-04- 
719. Washington, D.C.: May 14, 2004. 

Customs Service Modernization: Automated Commercial Environment 
Progressing, but Further Acquisition Management Improvements Needed. GAO- 
03-406. Washington, D.C.: February 28, 2003. 

Customs Service Modernization: Third Expenditure Plan Meets Legislative 
Conditions, but Cost Estimating Improvements Needed. GAO-02-908. 
Washington, D.C.: August 9, 2002. 
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Attachment 2 
Related GAO Products 



Customs Service Modernization: Management Improvements Needed on High- 
Risk Automated Commercial Environment Project. GAO-02-545. Washington, 
D.C.: May 13, 2002. 

Customs Service Modernization: Results of Review of First Automated 
Commercial Environment Expenditure Plan. GAO-01-696. Washington, D.C.: 
June 5, 2001. 

Other 

Information Security: Homeland Security Needs to Enhance Effectiveness of Its 
Program. GAO-07-1003T. Washington, D.C.: June 20, 2007. 

Intellectual Property: Better Data Analysis and Integration Could Help U.S. 
Customs and Border Protection Improve Border Enforcement Efforts. GAO-07- 
735. Washington, D.C.: April 26, 2007. 

Cargo Container Inspections: Preliminary Observations on the Status of Efforts 
to Improve the Automated Targeting System. GAO-06-591T. Washington, D.C.: 
March 30, 2006. 
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U.S. Department of Homeland Security 

Washington, DC 20528 




October 10, 2007 



Mr. Randolph C. Hite 

Director, Information Technology Architecture 

and Systems Issues 
U.S. Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Mr. Hite: 

RE: Draft Report GAO-08-46, Information Technology: Improvements for 
Acquisition of Customs Trade Processing System Continue, but Further 
Efforts Needed to Avoid More Cost and Schedule Shortfalls 
(GAO Job Code 310634) 

The Department of Homeland Security (DHS) appreciates the opportunity to review and 
comment on the draft report referenced above. Consistent with the Fiscal Year 2007 
DHS appropriations act, DHS is to develop and submit an expenditure plan for the 
Automated Commercial Environment (ACE) that satisfies certain conditions, including 
being reviewed by the U.S. Government Accountability Office (GAO). The report 
correctly notes that the expenditure plan satisfies many of the legislative conditions 
specified in the act and that the Department continues to make progress. 

GAO recognizes that we have implemented eight GAO recommendations made during 
the last four years and that seven other recommendations made during that time are in the 
process of being implemented. Department and U.S. Customs and Border Protection 
(CBP) officials are committed to fully addressing these remaining open recommendations 
regarding the ACE accountability framework; measures and associated program 
management metrics for determining success of improvements; minimizing the degree of 
overlap and concurrency across ACE releases and mitigating associated risks; meeting 
the legislative conditions regarding a privacy impact assessment and ensuring 
architectural alignment; implementing key human capital management practices; 
managing human capital needs; and reporting progress implementing GAO 
recommendations to Congress quarterly. 

GAO recommends three actions involving program information disclosure to the House 
and Senate Appropriations Committees designed to further strengthen ACE management 
and accountability for ACE performance and results. U.S. Customs and Border 
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Protection officials agree with the recommendations. The new recommendations and 
CBP's planned corrective actions follow. 

Recommendation 1 : [Future quarterly reports disclose] the risks and associated mitigation 
strategies of not having fully satisfied the expenditure plan legislative conditions and not 
having completed implementation of all prior GAO recommendations. 

Response: GAO concluded that CBP satisfied four of six legislative conditions but only 
partially satisfied the following two conditions: 

(1) Meet the capital planning and investment control review requirements established 
by the Office of Management and Budget (OMB), including OMB Circular A-l 1, 
Preparation, Submission, and Execution of the Budget, Part 7 [Planning, 
Budgeting, Acquisition and Management of Capital Assets]. The risk of not 
completely satisfying this requirement as part of the expenditure plan is minimal 
as CBP does meet OMB's requirements, but must better demonstrate compliance 
in future expenditure plan submissions. Full alignment to OMB's investment 
management requirements can be demonstrated via the Office of Management 
and Budget Exhibit 300 form [ACE Exhibit 300] that will be attached to all future 
ACE Expenditure Plan submissions. 

(2) Comply with the DHS enterprise architecture. DHS officials determined in May 
2007 that all required ACE products and technologies were aligned with the DHS 
technical reference model and that ACE was thus aligned with the DHS enterprise 
architecture. CBP agrees with GAO that a compliance determination is not a one- 
time event but a series of determinations that occurs throughout an investment's 
life cycle. CBP will continue to address DHS architectural requirements as DHS 
evolves its alignment methodology and criteria to meet GAO expectations. 

Recommendation 2 : [Future quarterly reports disclose] the status and impacts on the 
program's estimated cost and schedule and lessons learned from ongoing efforts to 
redefine requirements and to implement a different commercial off-the-shelf product than 
originally selected. 

Response : ACE program officials intend to disclose this information in future quarterly 
reports. 

Recommendation 3 : [Future quarterly reports disclose] the program's plans and actions 
for improving ACE risk management and its current inventory of program risks, 
including their associated mitigation strategies and the status of the strategies' 
implementation. 

Response : Officials responsible for ACE plan to strengthen risk management by: 

(1) Undertaking a new round of risk identification efforts to include the appropriate 
contractor technical and management personnel as well as government personnel. 
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The first session was held on September 14, 2007. After these identification 
sessions, the list of risks will be analyzed and prioritized. A response strategy 
will be developed for each risk. 

(2) The risks will be categorized and linked to the 19 risk categories in the OMB 300 
document for information technology programs. This effort will ensure that the 
critical areas of risk are covered. 

(3) Reports from the risk management software will be made available to the 
appropriate oversight personnel. The reports will include the mitigation strategy 
and status of its implementation. 

The foregoing actions should enable CBP to ensure that future Congressional reports 
provide complete and current information regarding program risks and attendant 
mitigation strategies, as well as the status of efforts to mitigate these risks. 

Sincerely, 



Steven J. Pecinovsky 
Director 

Departmental GAO/OIG Liaison Office 



MMcP 
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GAO's Mission 



Obtaining Copies of 
GAO Reports and 
Testimony 

Order by Mail or Phone 



The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting its 
constitutional responsibilities and to help improve the performance and 
accountability of the federal government for the American people. GAO 
examines the use of public funds; evaluates federal programs and policies; 
and provides analyses, recommendations, and other assistance to help 
Congress make informed oversight, policy, and funding decisions. GAO's 
commitment to good government is reflected in its core values of 
accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts 
newly released reports, testimony, and correspondence on its Web site. To 
have GAO e-mail you a list of newly posted products every afternoon, go 
to www.gao.gov and select "E-mail Updates." 

The first copy of each printed report is free. Additional copies are $2 each. 
A check or money order should be made out to the Superintendent of 
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. Orders 
should be sent to: 

U.S. Government Accountability Office 
441 G Street NW, Room LM 
Washington, DC 20548 



To order by Phone: Voice: 
TDD: 
Fax: 



(202) 512-6000 
(202) 512-2537 
(202) 512-6061 



To Report Fraud, 
Waste, and Abuse in 
Federal Programs 

Congressional 
Relations 

Public Affairs 



Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 
E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, jarmonG@gao.gov, (202) 512-4400 
U.S. Government Accountability Office, 441 G Street NW, Room 7125 
Washington, DC 20548 

Chuck Young, Managing Director, youngcl@gao.gov, (202) 512-4800 
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548 
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